Skip to contentSkip to conversation

GitLab Custom Runners and Jobs (was: QEMU Gating CI)

PATCH Superseded Cleber Rosa 8 participants 46 messages Jun 8, 2021 – Jun 30, 2021

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover GitLab Custom Runners and Jobs (was: QEMU Gating CI)

TL;DR: this should allow the QEMU maintainer to push to the staging branch, and have custom jobs running on the project's aarch64 and s390x machines. Jobs in this version are allowed to fail, to allow for the inclusion of the novel machines/jobs without CI disruption. Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

At this point, there are two specific runners (an aarch64 and an s390x) registered with GitLab, at https://gitlab.com/qemu-project, currently setup to the "qemu" repository. To make it extra clear, the following points deserve notice:

- This work dealt with two different QEMU project machines:

    I) s390x running Ubuntu 18.04

II) aarch64 running Ubuntu 20.04

- All CI jobs introduced here are allowed to fail. It should have no impact on the overall PASS/FAIL result of the pipeline.

- Both machines have already been completely configured used the code on this series. No further action (besides pushing this code to the staging branch) is needed to have CI jobs executed on them.

- The actual CI jobs to be executed are defined in this series, and map as best as possible the tests run by Peter Maydell on the staging branch. Peter, given the time delta since the beginning of this work, it may be needed to further tweak them.

- The actual results of CI jobs run from these definitions are probably more fluid than the results from the CI jobs run on the shared runners. Both changes to the code being tested, and the conditions/resources of the machine should be taken into account.

- A pipeline, and some of the jobs, can be seen on the links bellow. Besides successful jobs, it also contains jobs not started (set to manual, with the specific reasons noted later on the changes section):

* https://gitlab.com/cleber.gnu/qemu/-/pipelines/316471691
- ubuntu-18.04-s390x-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698118
- ubuntu-20.04-aarch64-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698124

Changes from v5:

- Moved include of ".gitlab-ci.d/custom-runners.yml" from ".gitlab-ci.yml" to ".gitlab-ci.d/qemu-project.yml"

- Changed git clone strategy from "submodule recursive" to "clone", to guarantee a fresh and clean repo on every job, and let QEMU handle eventual needed recursive submodule operations

- Require user to create an Ansible inventory file, based on the now provided "inventory.template" one. Previously, the "iventory" file itself was provided and users were asked to edit it.

- Registered runners will now be given a default set of tags, with their OS and architectures. This, besides automating another step, works around a "gitlab-runner register" command line issue with "--run-untagged=false" not being respected if no tags are given.

- Added conditional for installing either "clang" or "clang-10" to match the package name on different versions of Ubuntu.

- Changed Ubuntu 20.04 jobs to use clang-10 instead of clang.

- Defaults to not running untagged jobs when registering a gitlab runner.

- Added python3-sphinx-rtd-theme package installation to build-environment.yml, to match 73e6aec6522.

- Added genisoimage package installation to build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match 7e86e5d5ccc.

- Added liblttng-ust-dev package installation to build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match 8e9419b7902.

- Added libslirp-dev package installation to build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match 8e9419b7902.

- Added netcat-openbsd package installation to build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match c4cb1c9f2e1.

- Bumped gitlab-runner version to 13.12.0

- Use only gitlab-runner binaries from the official distribution (they are now also available for s390x, previously missing).

- Selection of the OS and architecture for the gitlab-runner binary is now done automatically. If necessary, architecture chosen can be influenced by editing the "ansible_to_gitlab_arch" values on vars.yml (after you've created one from the vars.yml.template).

- Marked job "ubuntu-18.04-s390x-clang" as manual, given that the latest executions are getting stuck with the last output being "Sanitizers are enabled ==> Not running the qemu-iotests."

- Marked job "ubuntu-18.04-s390x-notcg" as manual, given that the latest executions are getting stuck with the last tests being "tests/qtest/pxe-test" and "tests/qtest/boot-serial-test" which contains commands lines with "-accel kvm -accel tcg" and "-accel qtest". Obviously tcg is disabled and KVM is not available on the QEMU s390x machine, and the fallback to qtest gets the tests stuck.

- Marked job "ubuntu-20.04-aarch64-notcg" as manual due to build failures (FAILED: libqemu-aarch64-softmmu.fa.p/target_arm_psci.c.o).

- Marked job "ubuntu-20.04-aarch64-clang" as manual due to test failures:
../tests/qtest/libqtest.c:157: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0) ERROR qtest-aarch64/qom-test - too few tests run (expected 80, got 6)

- Cleared Daniel's R-B due to the number of changes in the corresponding patches.

Known issues:

- tests/qtest/qos-test has been seen stuck a number of times on both the s390x and aarch64 machines. Further investigation is needed. While a resolution is found, this should not affect the pipeline, given that all new jobs introduced here are currently allowed to fail.

Next development / discussion points:

- Test docker executor
- Test docker executor with podman
- Configuration of more accurate job timeouts
- Historical result analysis
- Promotion of jobs (removing the "allow failure" setting)

Changes from v4:

- Fixed typo in docs/devel/ci.rst, s/maintanance/maintenance/ (Thomas)
- Removed "[local]" group from inventory file (Erik)
- Removed sections from the playbooks which *would* be applied on hardware/OS that are currently not available to QEMU
- Removed duplicated "here" on documentation (Thomas)
- Moved description of current jobs, and possible direction of future jobs to the patch description (Thomas)
- Remove comments around "when" conditions (Andrea)
- Switch to always use explicit lists on "when" blocks (Andrea)
- Switch from using module "apt" to using generic action module "package", which involved adding a new task to update the apt cache (Andrea)
- Fix playbook indentation in the non-s390x package installation task (Andrea)
- Changed gitlab-runner tags examples from FreeBSD to Ubuntu, which is covered by jobs added on this version
- Fixed typo in commit message s/s390/s390x/ (Phil)
- Allow all custom-runner jobs to fail at this time
- Cleared "Reviewed-by" in one patch due to large changes

Changes requested in v4 but *not* seen here due to sections of the playbook being removed:

- Replace SDL-devel for SDL2-devel on CentOS, according to 5ed7ca3 (Thomas)
- Correct missing step 10 on the FreeBSD gitlab-runner installation instructions (Erik)

Changes from v3:

- Applied changes to match <[email protected]>, that is, added ninja-build to "build-environment.yml" list of packages and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners", given that the other jobs running on shared runners are also considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
* update -> up to date (Erik)
* a different set of CI jobs -> different CI jobs (Erik)
* Pull requests will only be merged -> code will only be merged (Stefan)
* Setup -> set up (Stefan)
* them -> they (Stefan)
* the -> where the (Stefan)
* dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that no external variable is used there

- Updated package list in "build-environment.yml" from current dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30), and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts" into (Phillipe):
1) Basic documentation and configuration (gitlab-ci.yml) placeholder
2) Playbooks for setting up a build environment
3) Playbooks for setting up gitlab-runner
4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine build and test.

- Set custom jobs to not depend on any other job, so they can start right away.

- Set rules for starting jobs so that all pushing to any branch that start with name "staging". This allows the project maintainer to use the "push to staging" workflow, while also allowing others to generate similar jobs. If this project has configured custom runners, the jobs will run, if not, the pipeline will be marked as "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in ".gitlab-ci.yml", assumes that the additional jobs on the staging branch running on the freely available gitlab shared runner are positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to simply 18.04 (assumes no diverse minor levels will be used or specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
Jobs based on custom runners: documentation and configuration
placeholder
Jobs based on custom runners: build environment docs and playbook Jobs based on custom runners: docs and gitlab-runner setup playbook Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 222 +++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml          |   1 +
 docs/devel/ci.rst                      | 115 +++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   1 +
 scripts/ci/setup/build-environment.yml |  98 +++++++++++
 scripts/ci/setup/gitlab-runner.yml     |  61 +++++++
 scripts/ci/setup/inventory.template    |   1 +
 scripts/ci/setup/vars.yml.template     |  12 ++
 9 files changed, 512 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory.template
 create mode 100644 scripts/ci/setup/vars.yml.template
Signature · 1 line
2.25.4
#1 lore raw

PATCH v6 1/4 Jobs based on custom runners: documentation and configuration placeholder

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true").

---
 .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
 .gitlab-ci.d/qemu-project.yml   |  1 +
 docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 44 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
.gitlab-ci.d/custom-runners.yml new file +14 -0
new file mode 100644
index 0000000000..a07b27384c
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+  GIT_STRATEGY: clone
.gitlab-ci.d/qemu-project.yml +1 -0
index 64cb2ba1da..dde8270301 100644
@@ -9,3 +9,4 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
docs/devel/ci.rst new file +28 -0
new file mode 100644
index 0000000000..585b7bf4b8
@@ -0,0 +1,28 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
docs/devel/index.rst +1 -0
index 791925dcda..c9a02e786e 100644
@@ -27,6 +27,7 @@ Contents:
    migration
    atomics
    stable-process
+   ci
    qtest
    decodetree
    secure-coding-practices
Signature · 1 line
2.25.4
Reviewed-by on v6 1/4
#2 lore raw

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

Quoted 12 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true").

Quoted 8 lines
---
  .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
  .gitlab-ci.d/qemu-project.yml   |  1 +
  docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
  docs/devel/index.rst            |  1 +
  4 files changed, 44 insertions(+)

create mode 100644 .gitlab-ci.d/custom-runners.yml create mode 100644 docs/devel/ci.rst

Quoted 76 lines
.gitlab-ci.d/custom-runners.yml new file +14 -0
new file mode 100644
index 0000000000..a07b27384c
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+  GIT_STRATEGY: clone
.gitlab-ci.d/qemu-project.yml +1 -0
index 64cb2ba1da..dde8270301 100644
@@ -9,3 +9,4 @@ include:
    - local: '/.gitlab-ci.d/crossbuilds.yml'
    - local: '/.gitlab-ci.d/buildtest.yml'
    - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
docs/devel/ci.rst new file +28 -0
new file mode 100644
index 0000000000..585b7bf4b8
@@ -0,0 +1,28 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
docs/devel/index.rst +1 -0
index 791925dcda..c9a02e786e 100644
@@ -27,6 +27,7 @@ Contents:
     migration
     atomics
     stable-process
+   ci
     qtest
     decodetree
     secure-coding-practices
Reviewed-by on v6 1/4
#3 lore raw

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

In reply to Cleber Rosa
Quoted 5 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners.

Quoted 7 lines

This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true").

Signature · 1 line
Alex Bennée
Reviewed-by on v6 1/4
#4 lore raw

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

In reply to Cleber Rosa
Quoted 31 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true").

---
  .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
  .gitlab-ci.d/qemu-project.yml   |  1 +
  docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
  docs/devel/index.rst            |  1 +
  4 files changed, 44 insertions(+)

create mode 100644 .gitlab-ci.d/custom-runners.yml create mode 100644 docs/devel/ci.rst

.gitlab-ci.d/custom-runners.yml new file +4 -0
new file mode 100644
index 0000000000..a07b27384c
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.

.gitlab-ci.d/custom-runners.yml line 14

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml new file +10 -0
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+  GIT_STRATEGY: clone

I assume this now does not influence the shared runners anymore, now that the file is handled as last entry in the list below? Or is this a global setting that will also be used for the shared runners? Anyway, let's keep it for now, we can still try to improve later...

.gitlab-ci.d/qemu-project.yml include: · line 12

Quoted 9 lines
.gitlab-ci.d/qemu-project.yml +1 -0
index 64cb2ba1da..dde8270301 100644
@@ -9,3 +9,4 @@ include:
    - local: '/.gitlab-ci.d/crossbuilds.yml'
    - local: '/.gitlab-ci.d/buildtest.yml'
    - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
Reviewed-by on v6 1/4
#5 lore raw

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

In reply to Cleber Rosa
Quoted 13 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true").

Quoted 9 lines
---
 .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
 .gitlab-ci.d/qemu-project.yml   |  1 +
 docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 44 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
#6 lore raw

PATCH v6 2/4 Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +30 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +98 -0

98 lines not on this page · raw message

scripts/ci/setup/inventory.template new file +1 -0
new file mode 100644
index 0000000000..2fbb50c4a8
@@ -0,0 +1 @@
+localhost
Signature · 1 line
2.25.4
#7 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Hi,

Quoted 22 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
  docs/devel/ci.rst                      | 30 ++++++++
  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
  scripts/ci/setup/inventory.template    |  1 +
  3 files changed, 129 insertions(+)

create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template

docs/devel/ci.rst +0 -0
index 585b7bf4b8..35c6b5e269 100644

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 35

Quoted 10 lines
docs/devel/ci.rst +7 -0
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
  The GitLab CI jobs definition for the custom runners are located under::
 
    .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::

Missing to mention the template file.

Quoted 29 lines
docs/devel/ci.rst +23 -0
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +0 -0
new file mode 100644
index 0000000000..664f2f0519

scripts/ci/setup/build-environment.yml line 3

Quoted 3 lines
scripts/ci/setup/build-environment.yml new file +3 -0
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all

You will need to "become: yes" if the login user is not privileged, right?

Or mention on the documentation how the user should configure the connection for privileged login.

About privilege escalation with Ansible: https://docs.ansible.com/ansible/latest/user_guide/become.html

scripts/ci/setup/build-environment.yml line 4

Quoted 1 line
scripts/ci/setup/build-environment.yml new file +1 -0
+  tasks:

Just a tip: you can put all those task under a block (https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html) so check if "ansible_facts['distribution'] == 'Ubuntu'" only once.

I reviewed the remain of the playbook; it looks good to me.

Quoted 101 lines
scripts/ci/setup/build-environment.yml new file +94 -0
+    - name: Update apt cache
+      apt:
+        update_cache: yes
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - ccache
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libzstd-dev
+          - make
+          - python3-yaml
+          - python3-sphinx
+          - python3-sphinx-rtd-theme
+          - ninja-build
+          - sparse
+          - xfslibs-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      package:
+        name:
+          - libspice-server-dev
+          - libxen-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] != 's390x'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - clang
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '18.04'
+
+    - name: Install basic packages to build QEMU on Ubuntu 20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
+          - clang-10
+          - genisoimage
+          - liblttng-ust-dev
+          - libslirp-dev
+          - netcat-openbsd
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '20.04'
scripts/ci/setup/inventory.template new file +1 -0
new file mode 100644
index 0000000000..2fbb50c4a8
@@ -0,0 +1 @@
+localhost
#8 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 72 lines · Wainer dos Santos Moschetta

Hi,

Quoted 22 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
  docs/devel/ci.rst                      | 30 ++++++++
  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
  scripts/ci/setup/inventory.template    |  1 +
  3 files changed, 129 insertions(+)

create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template

docs/devel/ci.rst +0 -0
index 585b7bf4b8..35c6b5e269 100644

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 35

Quoted 10 lines
docs/devel/ci.rst +7 -0
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
  The GitLab CI jobs definition for the custom runners are located under::
 
    .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::

Missing to mention the template file.

Quoted 29 lines
docs/devel/ci.rst +23 -0
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +0 -0
new file mode 100644
index 0000000000..664f2f0519

scripts/ci/setup/build-environment.yml line 3

Quoted 3 lines
scripts/ci/setup/build-environment.yml new file +3 -0
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all

You will need to "become: yes" if the login user is not privileged, right?

Quoted 2 lines

Or mention on the documentation how the user should configure the connection for privileged login.

As this will vary from system to system, I think it is worth mentioning in the documentation it can be configured in the inventory file, adding the variable ansible_become=yes and ansible_become_password= if password is needed to sudo.

#9 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 72 lines · Willian Rampazzo
Quoted 70 lines · Wainer dos Santos Moschetta

Hi,

Quoted 22 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
   docs/devel/ci.rst                      | 30 ++++++++
   scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
   scripts/ci/setup/inventory.template    |  1 +
   3 files changed, 129 insertions(+)

create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template

docs/devel/ci.rst +0 -0
index 585b7bf4b8..35c6b5e269 100644

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 35

Quoted 10 lines
docs/devel/ci.rst +7 -0
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
   The GitLab CI jobs definition for the custom runners are located under::
 
     .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::

Missing to mention the template file.

Quoted 29 lines
docs/devel/ci.rst +23 -0
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +0 -0
new file mode 100644
index 0000000000..664f2f0519

scripts/ci/setup/build-environment.yml line 3

Quoted 3 lines
scripts/ci/setup/build-environment.yml new file +3 -0
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all

You will need to "become: yes" if the login user is not privileged, right?

Quoted 7 lines
Quoted 2 lines

Or mention on the documentation how the user should configure the connection for privileged login.

As this will vary from system to system, I think it is worth mentioning in the documentation it can be configured in the inventory file, adding the variable ansible_become=yes and ansible_become_password= if password is needed to sudo.

Agreed.  But I'll put a mention to the ansible-playbook command line options instead. I believe our savvy users will be able to decide whether to use the command line options, the inventory options, or even temporary changes to the playbook.

Thanks!

- Cleber.

#10 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Wainer dos Santos Moschetta
Quoted 63 lines · Wainer dos Santos Moschetta

Hi,

Quoted 22 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
  docs/devel/ci.rst                      | 30 ++++++++
  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
  scripts/ci/setup/inventory.template    |  1 +
  3 files changed, 129 insertions(+)

create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template

docs/devel/ci.rst +0 -0
index 585b7bf4b8..35c6b5e269 100644

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 35

Quoted 10 lines
docs/devel/ci.rst +7 -0
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
  The GitLab CI jobs definition for the custom runners are located under::
 
    .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::

Missing to mention the template file.

Quoted 26 lines
docs/devel/ci.rst +23 -0
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +0 -0
new file mode 100644
index 0000000000..664f2f0519
Quoted 9 lines

scripts/ci/setup/build-environment.yml line 3

Quoted 6 lines
scripts/ci/setup/build-environment.yml new file +3 -0
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all

You will need to "become: yes" if the login user is not privileged, right?

Hi Wainer,

Yes, pretty much all of the actions here need a privileged user. I'm not sure how much of Ansible basics we should be replicating here. In my experience, users running "ansible-playbook" will resort to a combination of --become, --become-method, --ask-become-pass, etc, if needed.

Quoted 3 lines

Or mention on the documentation how the user should configure the connection for privileged login.

Yes, I think it's better to just put a note/reference, instead of inundating the user and the playbook with possibly unnecessary configurations and details. I've added the following to the documentation:

Please note that most of the tasks in the playbook require superuser privileges, such as those from the ``root`` account or those obtained by ``sudo``. If necessary, please refer to ``ansible-playbook`` options such as ``--become``, ``--become-method``, ``--become-user`` and ``--ask-become-pass``.

Let me know if that sounds good to you.

Quoted 9 lines

About privilege escalation with Ansible:
https://docs.ansible.com/ansible/latest/user_guide/become.html

scripts/ci/setup/build-environment.yml line 4

Quoted 1 line
scripts/ci/setup/build-environment.yml new file +1 -0
+  tasks:

Just a tip: you can put all those task under a block
(https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html)
so check if "ansible_facts['distribution'] == 'Ubuntu'" only once.

There will be other non-Ubuntu tasks added later (such as in the CentOS Stream 8 series), so I think it's better to keep those separate. Also, the Ansible version check which Alex suggested is not limited to Ubuntu.

Quoted 2 lines

I reviewed the remain of the playbook; it looks good to me.

Thanks!
- Cleber.

#11 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa
Quoted 45 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +20 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory build-environment.yml

PLAY [Installation of basic packages to build QEMU] *************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [aarch64.ci.qemu.org]
ok: [hackbox-ubuntu-2004]

TASK [Update apt cache] *****************************************************************************************************************************************************
fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update apt cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update apt cache\n ^ here\n"}

          to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

PLAY RECAP ******************************************************************************************************************************************************************

  aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
  hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1
Signature · 1 line
Alex Bennée
#12 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 92 lines · Alex Bennée
Quoted 45 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +20 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1)
+ ansible-playbook -i inventory build-environment.yml

PLAY [Installation of basic packages to build QEMU]
*************************************************************************************************************************

TASK [Gathering Facts]
******************************************************************************************************************************************************
ok: [aarch64.ci.qemu.org]
ok: [hackbox-ubuntu-2004]

TASK [Update apt cache]
*****************************************************************************************************************************************************
fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check
'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'):
'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml':
line 5, column 7, but may\nbe elsewhere in the file depending on the exact
syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n -
name: Update apt cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check
'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'):
'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml':
line 5, column 7, but may\nbe elsewhere in the file depending on the exact
syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n -
name: Update apt cache\n ^ here\n"}

          to retry, use: --limit
Quoted 11 lines

@/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

PLAY RECAP
******************************************************************************************************************************************************************
aarch64.ci.qemu.org : ok=1 changed=0 unreachable=0
failed=1
hackbox-ubuntu-2004 : ok=1 changed=0 unreachable=0
failed=1

Hi Alex,

Thanks for checking this version out. It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004. Because I had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation on the "controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).

About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions. Could you please share the output of "ansible --version" on that machine?

Thanks again,
- Cleber.

[1] -
https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution

Quoted 4 lines

--
Alex Bennée

#13 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 95 lines · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <[email protected]> wrote:

Quoted 45 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +20 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory build-environment.yml

PLAY [Installation of basic packages to build QEMU]
*************************************************************************************************************************

TASK [Gathering Facts]
******************************************************************************************************************************************************

   ok: [aarch64.ci.qemu.org]
   ok: [hackbox-ubuntu-2004]

   TASK [Update apt cache]

*****************************************************************************************************************************************************

fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update apt
cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update apt
cache\n ^ here\n"}

           to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

PLAY RECAP
******************************************************************************************************************************************************************

   aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
   hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1
Quoted 8 lines

Hi Alex,

Thanks for checking this version out. It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004. Because I had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation on the "controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).

About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions. Could you please share the output of "ansible --version" on that machine?

15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version ansible 2.7.7
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

My machine is Debian stable (aka Buster)

Quoted 8 lines

Thanks again,
- Cleber.

[1] - https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution

--
Alex Bennée

Signature · 1 line
Alex Bennée
#14 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 143 lines · Alex Bennée
Quoted 1 line · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <[email protected]>

Quoted 7 lines · wrote:
Quoted 4 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

Quoted 8 lines
Quoted 8 lines

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98

++++++++++++++++++++++++++

Quoted 11 lines

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 26

Quoted 11 lines
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +0 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located

under::

Quoted 36 lines
Quoted 22 lines
docs/devel/ci.rst +20 -0
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

Quoted 1 line
   14:26:26 [alex@zen:~/l/q/s/c/setup]

review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory build-environment.yml

Quoted 3 lines

PLAY [Installation of basic packages to build QEMU]

*************************************************************************************************************************

Quoted 3 lines

TASK [Gathering Facts]

******************************************************************************************************************************************************

Quoted 6 lines
   ok: [aarch64.ci.qemu.org]
   ok: [hackbox-ubuntu-2004]

   TASK [Update apt cache]

*****************************************************************************************************************************************************

Quoted 2 lines
   fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional

check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error

Quoted 1 line

was: error while evaluating conditional (ansible_facts['distribution']

== 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe

Quoted 1 line

error appears to have been in

'/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe

Quoted 1 line

elsewhere in the file depending on the exact syntax problem.\n\nThe

offending line appears to be:\n\n tasks:\n - name: Update apt

Quoted 2 lines

cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional

check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error

Quoted 1 line

was: error while evaluating conditional (ansible_facts['distribution']

== 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe

Quoted 1 line

error appears to have been in

'/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe

Quoted 1 line

elsewhere in the file depending on the exact syntax problem.\n\nThe

offending line appears to be:\n\n tasks:\n - name: Update apt

Quoted 2 lines

cache\n ^ here\n"}

           to retry, use: --limit

@/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

Quoted 3 lines

PLAY RECAP

******************************************************************************************************************************************************************

Quoted 2 lines

aarch64.ci.qemu.org : ok=1 changed=0 unreachable=0

failed=1

Quoted 1 line

hackbox-ubuntu-2004 : ok=1 changed=0 unreachable=0

failed=1

Quoted 4 lines

Hi Alex,

Thanks for checking this version out. It looks like this is similar to

what happened to you during v5 and hackmox-ubuntu-2004. Because I

Quoted 1 line

had no issues running the playbook against aarch64.ci.qemu.org, I am

now betting that this is an issue with the ansible installation on the

Quoted 1 line

"controller" machine, that is, the one that drives the playbook

execution (the "zen" host, according to your output).

Quoted 2 lines

About the error, the syntax itself is correct[1], but I bet there may be

differences between ansible versions. Could you please share the

Quoted 1 line

output of "ansible --version" on that machine?

15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4
+ ansible --version
ansible 2.7.7
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/alex/.ansible/plugins/modules',
'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible executable location = /usr/bin/ansible
python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

Quoted 3 lines

My machine is Debian stable (aka Buster)

Hi Alex,

I found out, the hard way, that one will need ansible 2.8.x. I'll make a note about that in the documentation. Long version:

I was able to replicate this on Debian stable, using the exact same versions as you did. Then I tried the latest ansible 2.7.18 (the latest in the 2.7.x series), installed with "pip install ansible==2.7.18 --user". Still, the exact same issue.

Then, still on Debian stable, I tried 2.8.20 (installed with "pip install ansible==2.8.20 --user"), and the playbook execution completed successfully.

I went through the 2.7x. and 2.8.x changelogs, and I could not spot one clear entry about the change in behavior though.

Thanks,
- Cleber.

Quoted 15 lines
Quoted 5 lines

Thanks again,
- Cleber.

[1] -

https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution

Quoted 3 lines

--
Alex Bennée

--
Alex Bennée

#15 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 130 lines · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 11:26 AM Alex Bennée <[email protected]> wrote:

Quoted 63 lines · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <[email protected]> wrote:

Quoted 45 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +20 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

Quoted 1 line
   14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory

build-environment.yml

Quoted 6 lines

PLAY [Installation of basic packages to build QEMU]
*************************************************************************************************************************

TASK [Gathering Facts]

******************************************************************************************************************************************************

Quoted 6 lines
 
   ok: [aarch64.ci.qemu.org]
   ok: [hackbox-ubuntu-2004]

   TASK [Update apt cache]

*****************************************************************************************************************************************************

Quoted 2 lines

fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The

error

Quoted 3 lines

was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update

apt

Quoted 2 lines

cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The

error

Quoted 3 lines

was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update

apt

Quoted 5 lines

cache\n ^ here\n"}

           to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

PLAY RECAP

******************************************************************************************************************************************************************

Quoted 7 lines
 
   aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
   hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1

Hi Alex,

Thanks for checking this version out. It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004.

Because I

Quoted 1 line

had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation

on the

Quoted 3 lines

"controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).

About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions. Could you please share

the

Quoted 1 line

output of "ansible --version" on that machine?

15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version ansible 2.7.7
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

Quoted 6 lines

My machine is Debian stable (aka Buster)

Hi Alex,

I found out, the hard way, that one will need ansible 2.8.x. I'll make a note about that in the documentation. Long version:

OK - yes please and I'll test with a local installed version. Is it possible for the playbook to error out with a useful warning if the version of ansible isn't new enough?

Quoted 23 lines

I was able to replicate this on Debian stable, using the exact same versions as you did. Then I tried the latest ansible 2.7.18 (the latest in the 2.7.x series), installed with "pip install ansible==2.7.18 --user". Still, the exact same issue.

Then, still on Debian stable, I tried 2.8.20 (installed with "pip install ansible==2.8.20 --user"), and the playbook execution completed successfully.

I went through the 2.7x. and 2.8.x changelogs, and I could not spot one clear entry about the change in behavior though.

Thanks,
- Cleber.

Quoted 8 lines

Thanks again,
- Cleber.

[1] - https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution

--
Alex Bennée

--
Alex Bennée

Signature · 1 line
Alex Bennée
#16 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 134 lines · Alex Bennée
Quoted 132 lines · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 11:26 AM Alex Bennée <[email protected]> wrote:

Quoted 63 lines · Cleber Rosa Junior

On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <[email protected]> wrote:

Quoted 45 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +20 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 58

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I have up and running as a VM and I got a failure when checking facts:

Quoted 1 line
   14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory

build-environment.yml

Quoted 6 lines

PLAY [Installation of basic packages to build QEMU]
*************************************************************************************************************************

TASK [Gathering Facts]

******************************************************************************************************************************************************

Quoted 6 lines
   ok: [aarch64.ci.qemu.org]
   ok: [hackbox-ubuntu-2004]

   TASK [Update apt cache]

*****************************************************************************************************************************************************

Quoted 2 lines

fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The

error

Quoted 3 lines

was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update

apt

Quoted 2 lines

cache\n ^ here\n"}
fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The

error

Quoted 3 lines

was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Update

apt

Quoted 5 lines

cache\n ^ here\n"}

           to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

PLAY RECAP

******************************************************************************************************************************************************************

Quoted 7 lines
   aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
   hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1

Hi Alex,

Thanks for checking this version out. It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004.

Because I

Quoted 1 line

had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation

on the

Quoted 3 lines

"controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).

About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions. Could you please share

the

Quoted 1 line

output of "ansible --version" on that machine?

  15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version
  ansible 2.7.7
    config file = /etc/ansible/ansible.cfg
    configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python3/dist-packages/ansible
    executable location = /usr/bin/ansible
    python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

My machine is Debian stable (aka Buster)

Quoted 7 lines
Quoted 4 lines

Hi Alex,

I found out, the hard way, that one will need ansible 2.8.x. I'll make a note about that in the documentation. Long version:

OK - yes please and I'll test with a local installed version. Is it possible for the playbook to error out with a useful warning if the version of ansible isn't new enough?

Hi Alex,

Yes, I'll add the version check on v7.  The planned error message will be:

"Unsuitable ansible version, please use version 2.8.0 or later"

- Cleber.

Reviewed-by Tested-by on v6 2/4
#17 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa
Quoted 34 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template
docs/devel/ci.rst +8 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 46

Quoted 10 lines
docs/devel/ci.rst +10 -0
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+

As I mentioned to Wainer, my suggestion is to include a comment about ansible_become=yes and
ansible_become_password= variables as some systems may need it.

Quoted 18 lines
docs/devel/ci.rst +12 -0
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +0 -0
new file mode 100644
index 0000000000..664f2f0519

scripts/ci/setup/build-environment.yml line 7

Quoted 7 lines
scripts/ci/setup/build-environment.yml new file +7 -0
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:
+    - name: Update apt cache
+      apt:
+        update_cache: yes

On a freshly installed Ubuntu 20.04, the script failed for me with dependency messages on the apt side. After I updated the packages on the system, the playbook worked without problems.

So, my suggestion is to add the "update = yes" here, or add a note in the documentation asking the user to update the system before running the playbook.

Except for the above comment and Wainer's comments, it looks good to me. With these changes:

#18 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

...

Quoted 7 lines

On a freshly installed Ubuntu 20.04, the script failed for me with dependency messages on the apt side. After I updated the packages on the system, the playbook worked without problems.

So, my suggestion is to add the "update = yes" here, or add a note in the documentation asking the user to update the system before running the playbook.

I'd advocate for always updating the system automatically rather than putting it into the documentation and relying on users actually doing it.

Erik

#19 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 5 lines · Erik Skultety

...

Quoted 4 lines

On a freshly installed Ubuntu 20.04, the script failed for me with dependency messages on the apt side. After I updated the packages on the system, the playbook worked without problems.

Quoted 8 lines
Quoted 3 lines

So, my suggestion is to add the "update = yes" here, or add a note in the documentation asking the user to update the system before running the playbook.

I'd advocate for always updating the system automatically rather than putting it into the documentation and relying on users actually doing it.

Erik

ACK, I'm adding that to v7.

Thanks,

- Cleber.

#20 lore raw

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Willian Rampazzo
Quoted 39 lines · Willian Rampazzo
Quoted 38 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them.

---
  docs/devel/ci.rst                      | 30 ++++++++
  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
  scripts/ci/setup/inventory.template    |  1 +
  3 files changed, 129 insertions(+)

create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template

docs/devel/ci.rst +13 -0
index 585b7bf4b8..35c6b5e269 100644
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
  The GitLab CI jobs definition for the custom runners are located under::
 
    .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
Quoted 8 lines

docs/devel/ci.rst gitlab-runner, is called a "custom runner". · line 46

Quoted 5 lines
docs/devel/ci.rst +5 -0
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+

As I mentioned to Wainer, my suggestion is to include a comment about ansible_become=yes and
ansible_become_password= variables as some systems may need it.

OK.  I mentioned in a previous reply that I'll plan to do this slightly different, that is, to mention the ansible-playbook command line options.  Let me know if that works for you.

Quoted 20 lines
Quoted 20 lines
docs/devel/ci.rst +12 -0
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +2 -0
new file mode 100644
index 0000000000..664f2f0519
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
Quoted 8 lines

scripts/ci/setup/build-environment.yml line 7

Quoted 5 lines
scripts/ci/setup/build-environment.yml new file +5 -0
+  hosts: all
+  tasks:
+    - name: Update apt cache
+      apt:
+        update_cache: yes

On a freshly installed Ubuntu 20.04, the script failed for me with dependency messages on the apt side. After I updated the packages on the system, the playbook worked without problems.

I know it's too late now, but it would have been helpful to document the exact error, at the very least here in this thread. I tried to replicate with an also fresh install, but resulted in:

PLAY RECAP ************************************************************************************************************************************** 192.168.122.19             : ok=6    changed=4    unreachable=0 failed=0    skipped=1    rescued=0    ignored=0

A second run resulted in:

PLAY [Installation of basic packages to build QEMU] *********************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************
ok: [192.168.122.19]

TASK [Check for suitable ansible version] *******************************************************************************************************
ok: [192.168.122.19] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Update apt cache] *************************************************************************************************************************
ok: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 18.04/20.04] *******************************************************************************
ok: [192.168.122.19]

TASK [Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x] ************************************************************************
ok: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 18.04] *************************************************************************************
skipping: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 20.04] *************************************************************************************
ok: [192.168.122.19]

PLAY RECAP ************************************************************************************************************************************** 192.168.122.19             : ok=6    changed=0    unreachable=0 failed=0    skipped=1    rescued=0    ignored=0

Quoted 3 lines

So, my suggestion is to add the "update = yes" here, or add a note in the documentation asking the user to update the system before running the playbook.

IIUC, you mean an "upgrade: yes" on the "Update apt cache" task, right?  Something like:

-    - name: Update apt cache
+    - name: Update apt cache / upgrade packages via apt apt:
         update_cache: yes
+        upgrade: yes

I have mixed feelings about going beyond installing/updating the packages we care about.  On one hand, it's a good sysadmin practice, but on the other hand, development/test machines are sometimes peculiar.  Since you and Erik are in favor of the updates, I'll change that and wait for feedback on v7.

So, besides the change to the playbook, I'm adding to the docs:

The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will set up machines with the environment needed to perform builds and run QEMU tests.  This playbook consists on the installation of various required packages (and a general package update while at it).  It currently covers a number of different Linux distributions, but it can be expanded to cover other systems.

Quoted 6 lines

Except for the above comment and Wainer's comments, it looks good to me. With these changes:

Thanks for the review and test!  Given the amount of changes, I'll hold a bit to take you Rb/Tb until you see/test v7.

Thanks again!

- Cleber

#21 lore raw

PATCH v6 3/4 Jobs based on custom runners: docs and gitlab-runner setup playbook

In reply to Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
 docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 12 ++++++
 4 files changed, 131 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +57 -0
index 35c6b5e269..bbd89e54d7 100644
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +61 -0

61 lines not on this page · raw message

scripts/ci/setup/vars.yml.template new file +12 -0
new file mode 100644
index 0000000000..e48089761f
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
Signature · 1 line
2.25.4
#22 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Hi,

Quoted 50 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
  scripts/ci/setup/.gitignore        |  1 +
  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
  scripts/ci/setup/vars.yml.template | 12 ++++++
  4 files changed, 131 insertions(+)

create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/vars.yml.template

docs/devel/ci.rst +15 -0
index 35c6b5e269..bbd89e54d7 100644
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
    cd scripts/ci/setup
    ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+

docs/devel/ci.rst To run the playbook, execute:: · line 83

Quoted 10 lines
docs/devel/ci.rst +10 -0
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``

I think you can remove the information about the gitlab-running being not available for some systems.

Quoted 9 lines
docs/devel/ci.rst +9 -0
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+

docs/devel/ci.rst To run the playbook, execute:: · line 102

Quoted 10 lines
docs/devel/ci.rst +10 -0
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64

Also users no longer need manually create the tags.

Remaining of the file looks good to me.

Quoted 106 lines
docs/devel/ci.rst +13 -0
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +61 -0
new file mode 100644
index 0000000000..98dab92bb5
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Set the Operating System for gitlab-runner
+      set_fact:
+        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+    - debug:
+        msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+    - name: Set the architecture for gitlab-runner
+      set_fact:
+        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+    - debug:
+        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+    - name: Download the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
scripts/ci/setup/vars.yml.template new file +12 -0
new file mode 100644
index 0000000000..e48089761f
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
#23 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 58 lines · Wainer dos Santos Moschetta

Hi,

Quoted 54 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
  scripts/ci/setup/.gitignore        |  1 +
  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
  scripts/ci/setup/vars.yml.template | 12 ++++++
  4 files changed, 131 insertions(+)

create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/vars.yml.template

docs/devel/ci.rst +19 -0
index 35c6b5e269..bbd89e54d7 100644
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
    cd scripts/ci/setup
    ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
Quoted 8 lines

docs/devel/ci.rst To run the playbook, execute:: · line 83

Quoted 6 lines
docs/devel/ci.rst +6 -0
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``

I think you can remove the information about the gitlab-running being not available for some systems.

Good catch!

Quoted 13 lines
Quoted 13 lines
docs/devel/ci.rst +13 -0
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
Quoted 9 lines

docs/devel/ci.rst To run the playbook, execute:: · line 102

Quoted 6 lines
docs/devel/ci.rst +6 -0
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64

Also users no longer need manually create the tags.

Of course. But, given that this is a general rule, and there may be OS/architectures for which Ansible may produce tag names one would not expect, I'll ask the user to double check the tags. The text I'm proposing on v7 is:

Tags are very important as they are used to route specific jobs to specific types of runners, so it's a good idea to double check that the automatically created tags are consistent with the OS and architecture. For instance, an Ubuntu 20.04 aarch64 system should have tags set as::

ubuntu_20.04,aarch64

Quoted 2 lines

Remaining of the file looks good to me.

Thanks a lot!
- Cleber.

Reviewed-by Tested-by on v6 3/4
#24 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

In reply to Cleber Rosa
Quoted 36 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
 docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 12 ++++++
 4 files changed, 131 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +0 -0
index 35c6b5e269..bbd89e54d7 100644
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml

docs/devel/ci.rst To run the playbook, execute:: · line 68

Quoted 10 lines
docs/devel/ci.rst +10 -0
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then

* Settings (the gears like icon in the end of the left menu), then

I took some time to find it as it was hidden at the end of the scrolling.

docs/devel/ci.rst To run the playbook, execute:: · line 72

Quoted 4 lines
docs/devel/ci.rst +4 -0
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"

For me, it shows: "And this registration token:"

docs/devel/ci.rst To run the playbook, execute:: · line 81

Quoted 9 lines
docs/devel/ci.rst +9 -0
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable

If you keep this block (see comment from Wainer), s/avaiable/available/

Quoted 128 lines
docs/devel/ci.rst +34 -0
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +61 -0
new file mode 100644
index 0000000000..98dab92bb5
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Set the Operating System for gitlab-runner
+      set_fact:
+        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+    - debug:
+        msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+    - name: Set the architecture for gitlab-runner
+      set_fact:
+        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+    - debug:
+        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+    - name: Download the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
scripts/ci/setup/vars.yml.template new file +12 -1
new file mode 100644
index 0000000000..e48089761f
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
--
Quoted 2 lines

2.25.4

With my suggestions and Wainer's:

#25 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 36 lines · Willian Rampazzo
Quoted 34 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
 docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 12 ++++++
 4 files changed, 131 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +0 -0
index 35c6b5e269..bbd89e54d7 100644
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
Quoted 15 lines

docs/devel/ci.rst To run the playbook, execute:: · line 68

Quoted 12 lines
docs/devel/ci.rst +10 -0
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then

* Settings (the gears like icon in the end of the left menu), then

ACK. What about:

" * Settings (the gears-like icon at the bottom of the left hand side
vertical toolbar), then"

Quoted 9 lines

I took some time to find it as it was hidden at the end of the scrolling.

docs/devel/ci.rst To run the playbook, execute:: · line 72

Quoted 4 lines
docs/devel/ci.rst +4 -0
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"

For me, it shows: "And this registration token:"

Right, it must have changed. Updating it on v7.

Quoted 12 lines

docs/devel/ci.rst To run the playbook, execute:: · line 81

Quoted 9 lines
docs/devel/ci.rst +9 -0
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable

If you keep this block (see comment from Wainer), s/avaiable/available/

I'd rather remove it, but thanks for spotting the typo.

Quoted 136 lines
Quoted 128 lines
docs/devel/ci.rst +34 -0
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +61 -0
new file mode 100644
index 0000000000..98dab92bb5
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Set the Operating System for gitlab-runner
+      set_fact:
+        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+    - debug:
+        msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+    - name: Set the architecture for gitlab-runner
+      set_fact:
+        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+    - debug:
+        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+    - name: Download the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
scripts/ci/setup/vars.yml.template new file +12 -1
new file mode 100644
index 0000000000..e48089761f
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
--
Quoted 2 lines

2.25.4

With my suggestions and Wainer's:

#26 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

In reply to Cleber Rosa
Quoted 26 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
  scripts/ci/setup/.gitignore        |  1 +
  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
  scripts/ci/setup/vars.yml.template | 12 ++++++
  4 files changed, 131 insertions(+)

create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/vars.yml.template

[...]

scripts/ci/setup/.gitignore line 1

Quoted 8 lines
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file

Add a newline, please.

Quoted 6 lines
scripts/ci/setup/gitlab-runner.yml new file +0 -0
new file mode 100644
index 0000000000..98dab92bb5

scripts/ci/setup/gitlab-runner.yml line 9

Quoted 9 lines
scripts/ci/setup/gitlab-runner.yml new file +9 -0
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"

Could you please add a comment at the top of the file or name it differently so that it is clear from a quick glance that this is an ansible playbook? Poeple might later wonder otherwise...

Thomas

#27 lore raw

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 29 lines · Thomas Huth
Quoted 26 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely:

* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high.

---
  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
  scripts/ci/setup/.gitignore        |  1 +
  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
  scripts/ci/setup/vars.yml.template | 12 ++++++
  4 files changed, 131 insertions(+)

create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/vars.yml.template

[...]

Quoted 11 lines

scripts/ci/setup/.gitignore line 1

Quoted 8 lines
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file

Add a newline, please.

Sure! Thanks for spotting that.

Quoted 11 lines
Quoted 11 lines
scripts/ci/setup/gitlab-runner.yml new file +5 -0
new file mode 100644
index 0000000000..98dab92bb5
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
Quoted 9 lines

scripts/ci/setup/gitlab-runner.yml line 9

Quoted 4 lines
scripts/ci/setup/gitlab-runner.yml new file +4 -0
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"

Could you please add a comment at the top of the file or name it differently so that it is clear from a quick glance that this is an ansible playbook? Poeple might later wonder otherwise...

Good point. I'm adding a proper header with copyright / comment.

Quoted 2 lines

Thomas

Thanks for the review.
- Cleber.

#28 lore raw

PATCH v6 4/4 Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +208 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
 # guarantees a fresh repository on each job run.
 variables:
   GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
Signature · 1 line
2.25.4
#29 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Hi Alex, Stefan,

Quoted 2 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests.

AFAIK there is more hardware available to the project, so I'm wondering what happened to the rest, is it a deliberate choice to start small? What will happen with the rest, since we are wasting resources? Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for its system administration?

In case an entity manages hosting and maintenance, can the QEMU project share the power bill? Up to which amount?
Similar question if a sysadmin has to be paid.

If the QEMU project can't spend money on CI, what is expected from resource donators? Simply hardware + power (electricity) + network traffic? Also sysadmining and monitoring? Do we expect some kind of reliability on the data stored here or can we assume disposable / transient runners?
Should donators also provide storage? Do we have a minimum storage requirement?

Should we provide some guideline explaining any code might be run by our contributors on these runners and some security measures have to be taken / considered?

Sponsors usually expect some advertising to thanks them, and often regular reports on how their resources are used, else they might not renew their offer. Who should care to keep the relationship with sponsors?

Where is defined what belong to the project, who is responsible, who can request access to this hardware, what resource can be used?

More generically, what is the process for a private / corporate entity to register a runner to the project? (how did it work for this aarch64 and s390x one?)

What else am I missing?

Thanks,

Phil.

Quoted 7 lines

This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)
#30 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 9 lines · Philippe Mathieu-Daudé

Hi Alex, Stefan,

Quoted 2 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests.

AFAIK there is more hardware available to the project, so I'm wondering what happened to the rest, is it a deliberate choice to start small?

Hi Phil,

Yes, this series was deliberately focused on the first two machines owned and available to QEMU.

Quoted 2 lines

What will happen with the rest, since we are wasting resources?

The plan is to allow all machines (currently available and to-be available) to be connected as custom runners. This hopefully gets that started.

About "more hardware available to the project", there's one VM from fosshost which was made available not long ago, and which I set up even more recently, which could be used as a gitlab runner too. But, even though some new hardware resource is available (and wasted?), the human resources to maintain them have not been properly determined, so I believe it's a good decision to start with the machines that have been operational for long, and that already have to the best of my knowledge, people maintaining them.

I also see a "Debian unstable mips64el (Debian) @ cipunited.cn" registered as a runner, but I don't have extra information about it.

Besides that, I'll send another series shortly, that builds upon this series, and adds a Red Hat focused job, on a Red Hat managed machine. This should be what other entities should be capable of doing and allowed to do.

Quoted 3 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on a v7, I'd need the information on who is managing them.

Quoted 21 lines

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for its system administration?

In case an entity manages hosting and maintenance, can the QEMU project share the power bill? Up to which amount?
Similar question if a sysadmin has to be paid.

If the QEMU project can't spend money on CI, what is expected from resource donators? Simply hardware + power (electricity) + network traffic? Also sysadmining and monitoring? Do we expect some kind of reliability on the data stored here or can we assume disposable / transient runners?
Should donators also provide storage? Do we have a minimum storage requirement?

Should we provide some guideline explaining any code might be run by our contributors on these runners and some security measures have to be taken / considered?

Quoted 9 lines

Sponsors usually expect some advertising to thanks them, and often regular reports on how their resources are used, else they might not renew their offer. Who should care to keep the relationship with sponsors?

Where is defined what belong to the project, who is responsible, who can request access to this hardware, what resource can be used?

You obviously directed the question towards Alex and Stefan (rightfully so), so I won't attempt to answer these ones at this point.

Quoted 4 lines

More generically, what is the process for a private / corporate entity to register a runner to the project? (how did it work for this aarch64 and s390x one?)

The steps are listed on the documentation. Basically anyone with knowledge of the "registration token" can add new machines to GitLab as runners. For the two aarch64 and s390x, it was a matter of following the documentation steps which basically involve:

1) providing the hostname(s) in the inventory file
2) provide the "registration token" in the vars.yml file
3) running the playbooks

Quoted 4 lines

What else am I missing?

I think you're missing the answers to all your good questions :).

And I understand that are a lot of them (from everyone, including myself). The dilemma here is: should we activate the machines already available, and learn in practice, what's missing? I honestly believe we should.

Thanks,
- Cleber.

Quoted 13 lines

Thanks,

Phil.

Quoted 7 lines

This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)
#31 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Hi,

Quoted 87 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected] <mailto:[email protected]>> wrote:

Hi Alex, Stefan,

On 6/8/21 5:14 AM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.

AFAIK there is more hardware available to the project, so I'm wondering
what happened to the rest, is it a deliberate choice to start small?

Hi Phil,

Yes, this series was deliberately focused on the first two machines owned and available to QEMU.

What will happen with the rest, since we are wasting resources?

The plan is to allow all machines (currently available and to-be available) to be connected as custom runners. This hopefully gets that started.

About "more hardware available to the project", there's one VM from fosshost which was made available not long ago, and which I set up even more recently, which could be used as a gitlab runner too.  But, even though some new hardware resource is available (and wasted?), the human resources to maintain them have not been properly determined, so I believe it's a good decision to start with the machines that have been operational for long, and that already have to the best of my knowledge, people maintaining them.

I also see a "Debian unstable mips64el (Debian) @ cipunited.cn <http://cipunited.cn>" registered as a runner, but I don't have extra information about it.

Besides that, I'll send another series shortly, that builds upon this series, and adds a Red Hat focused job, on a Red Hat managed machine.  This should be what other entities should be capable of doing and allowed to do.

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

These are good questions, and I believe Alex can answer them about those two machines.  Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry.  This is still unresolved and feedback would be appreciated.  For me to propose a MAINTAINERS entry, say, on a v7, I'd need the information on who is managing them.

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for
its system administration?

In case an entity manages hosting and maintenance, can the QEMU project
share the power bill? Up to which amount?
Similar question if a sysadmin has to be paid.

If the QEMU project can't spend money on CI, what is expected from resource donators? Simply hardware + power (electricity) + network traffic? Also sysadmining and monitoring? Do we expect some kind of reliability on the data stored here or can we assume disposable / transient runners?
Should donators also provide storage? Do we have a minimum storage requirement?

Should we provide some guideline explaining any code might be run by our contributors on these runners and some security measures have to be taken / considered?

Sponsors usually expect some advertising to thanks them, and often regular reports on how their resources are used, else they might not renew their offer. Who should care to keep the relationship with sponsors?

Where is defined what belong to the project, who is responsible, who can
request access to this hardware, what resource can be used?

You obviously directed the question towards Alex and Stefan (rightfully so), so I won't attempt to answer these ones at this point.

More generically, what is the process for a private / corporate entity to register a runner to the project? (how did it work for this aarch64 and s390x one?)

The steps are listed on the documentation.  Basically anyone with knowledge of the "registration token" can add new machines to GitLab as runners.  For the two aarch64 and s390x, it was a matter of following the documentation steps which basically involve:

1) providing the hostname(s) in the inventory file
2) provide the "registration token" in the vars.yml file
3) running the playbooks

Quoted 6 lines

What else am I missing?

I think you're missing the answers to all your good questions :).

And I understand that are a lot of them (from everyone, including myself).  The dilemma here is: should we activate the machines already available, and learn in practice, what's missing?  I honestly believe we should.

IMHO we should merge the minimum possible to start using the existing machines, then address the questions (good questions, btw) raised by Philippe as needed.

Thanks!

- Wainer

Quoted 18 lines

Thanks,
- Clr.

Thanks,

Phil.

> This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose. >
> Signed-off-by: Cleber Rosa <[email protected] <mailto:[email protected]>>
> ---
>  .gitlab-ci.d/custom-runners.yml | 208
++++++++++++++++++++++++++++++++
>  1 file changed, 208 insertions(+)

#32 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 9 lines · Wainer dos Santos Moschetta
Quoted 4 lines

And I understand that are a lot of them (from everyone, including myself).  The dilemma here is: should we activate the machines already available, and learn in practice, what's missing?  I honestly believe we should.

IMHO we should merge the minimum possible to start using the existing machines, then address the questions (good questions, btw) raised by Philippe as needed.

That sounds good.

Does anyone want to volunteer to be the QEMU CI runners point of contact who is responsible for defining the process? Cleber's last email left a lot for Alex and me to define, but I would prefer it if someone who is not me takes this on since I'm already spread thin. Alex? Philippe? Cleber? Wainer?

Stefan

#33 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Wainer dos Santos Moschetta
Quoted 4 lines

And I understand that are a lot of them (from everyone, including myself). The dilemma here is: should we activate the machines already available, and learn in practice, what's missing? I honestly believe we should.

IMHO we should merge the minimum possible to start using the existing machines, then address the questions (good questions, btw) raised by Philippe as needed.

That sounds good.

Quoted 7 lines

Does anyone want to volunteer to be the QEMU CI runners point of contact who is responsible for defining the process? Cleber's last email left a lot for Alex and me to define, but I would prefer it if someone who is not me takes this on since I'm already spread thin. Alex? Philippe? Cleber? Wainer?

Stefan

Sure, I can do it.

I'll start by resurrecting the now ancient RFC that contains a lot of proposals with regards to the process, in light of the current status and developments:

https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html

IMO we can start with documentation (either at ci.rst, or preferably in the WIKI for now) for the process of onboarding new CI runners, and track it with issues in GitLab itself. I'll formalize those opinions.

Thanks,
- Cleber.

#34 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa Junior
Quoted 2 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected]> wrote:

Here are my thoughts. It's just my opinion based on experience running the old QEMU Buildbot infrastructure and interacting with our hosting providers. I'm not planning to actively work on the CI infrastructure, so if you have a strong opinion and want to do things differently yourself, feel free.

Quoted 4 lines
Quoted 3 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

Quoted 8 lines

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on a v7, I'd need the information on who is managing them.

Here is the wiki page that lists project resources (machines, etc):
https://wiki.qemu.org/AdminContacts

We can continue to use this page.

Quoted 4 lines
Quoted 4 lines

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for its system administration?

A document is needed that explains the process and the roles/responsibilities of the people involved.

QEMU doesn't have a physical presence and we currently don't have a way to host physical machines. We also probably shouldn't get involved in that because it has a high overhead and puts the responsibility on the project to maintain the hardware. There are hosting providers like OSUOSL that offer non-x86 architectures, so I don't think we need to deal with physical hardware even for other architectures. If someone needs their special hardware covered, let them act as the sponsor and sysadmin for that machine - they'll need to figure out where to host it.

Quoted 3 lines
Quoted 3 lines

In case an entity manages hosting and maintenance, can the QEMU project share the power bill? Up to which amount?
Similar question if a sysadmin has to be paid.

No, it's too complicated. QEMU does not have regular income that could be used for periodic expenses. We shouldn't spend time worrying about this unless there is a real need and plenty of funding.

Quoted 5 lines
Quoted 5 lines

If the QEMU project can't spend money on CI, what is expected from resource donators? Simply hardware + power (electricity) + network traffic? Also sysadmining and monitoring? Do we expect some kind of reliability on the data stored here or can we assume disposable / transient runners?

Sponsors provide access to internet-connected machines. They can designate a QEMU community volunteer to admin machines or they can admin the machine themselves.

Sysadmins deal with keeping the machine online (security, network traffic, monitoring, etc).

For simplicity it's best if sponsored machines are owned and paid for by the sponsor. Billing electricity, bandwidth, etc to QEMU will make things complicated and we don't have the admin infrastructure to support that.

Quoted 2 lines
Quoted 2 lines

Should donators also provide storage? Do we have a minimum storage requirement?

Sponsors provide storage. There is a minimum storage requirement that depends on the nature of CI jobs (I don't know what the exact amount is and it may change over time).

Quoted 2 lines
Quoted 2 lines

Where is defined what belong to the project, who is responsible, who can request access to this hardware, what resource can be used?

Machines belong to their sponsors, not to the QEMU project. They could go away in the future if the sponsor decides to withdraw them.

Only the sysadmin has ssh access to the machine. The CI system provides access to logs so that ssh access to machines is not necessary for QEMU developers. If ssh access is needed then the developer can ask the sysadmin for help.

Regarding resource usage, that's up to the sysadmin. If they want to apply resource limits to the CI environment they need to configure that.

Stefan

#35 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 24 lines · Stefan Hajnoczi
Quoted 2 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected]> wrote:

Here are my thoughts. It's just my opinion based on experience running the old QEMU Buildbot infrastructure and interacting with our hosting providers. I'm not planning to actively work on the CI infrastructure, so if you have a strong opinion and want to do things differently yourself, feel free.

Quoted 4 lines
Quoted 3 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

Quoted 8 lines

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on a v7, I'd need the information on who is managing them.

Quoted 5 lines

Here is the wiki page that lists project resources (machines, etc):
https://wiki.qemu.org/AdminContacts

We can continue to use this page.

ACK. I'm adding a note to the documentation.

Thanks,
- Cleber.

#36 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa Junior
Quoted 8 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected]> wrote:

Hi Alex, Stefan,

Quoted 2 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests.

<snip>

Quoted 10 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on
a v7, I'd need the information on who is managing them.

I can only talk about aarch64.ci.qemu.org which is a donated Equinix machine that comes from the WorksOnArm initiative. I applied for it on behalf of the QEMU project and we can have it for as long as it's useful.

I don't know if we need anything more that documenting the nominal contacts in:

https://wiki.qemu.org/AdminContacts

Quoted 4 lines

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for its system administration?

I think the project can only really work with donations out of someones data centre where they keep responsibility for the physical aspects of any machines including the ongoing hosting and running costs.

Signature · 1 line
Alex Bennée
#37 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 29 lines · Alex Bennée
Quoted 8 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected]> wrote:

Hi Alex, Stefan,

Quoted 2 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests.

<snip>

Quoted 10 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on
a v7, I'd need the information on who is managing them.

I can only talk about aarch64.ci.qemu.org which is a donated Equinix machine that comes from the WorksOnArm initiative. I applied for it on behalf of the QEMU project and we can have it for as long as it's useful.

Quoted 5 lines

I don't know if we need anything more that documenting the nominal contacts in:

https://wiki.qemu.org/AdminContacts

That's enough indeed, thanks. I'll follow up with a proposal about the expected duties of admins, which should be nothing but common sense.

Is there anyone that can speak for the s390x machine?

Quoted 9 lines
Quoted 4 lines

Is there a document explaining what are the steps to follow for an entity to donate / sponsor hardware? Where would it be stored, should this hw be shipped somewhere? What documentation should be provided for its system administration?

I think the project can only really work with donations out of someones data centre where they keep responsibility for the physical aspects of any machines including the ongoing hosting and running costs.

Agreed. Anything else is beyond what can be managed atm.

Quoted 3 lines

--
Alex Bennée

Thanks,
- Cleber.

#38 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 34 lines · Cleber Rosa
Quoted 32 lines · Alex Bennée
Quoted 8 lines · Cleber Rosa Junior

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <[email protected]> wrote:

Hi Alex, Stefan,

Quoted 2 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests.

<snip>

Quoted 10 lines

Who has access to what and should do what (setup)? How is this list of hw managed btw? Should there be some public visibility (i.e. Wiki)?

These are good questions, and I believe Alex can answer them about those two machines. Even though I hooked them up to GitLab, AFAICT he is the ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry. This is still unresolved and feedback would be appreciated. For me to propose a MAINTAINERS entry, say, on
a v7, I'd need the information on who is managing them.

I can only talk about aarch64.ci.qemu.org which is a donated Equinix machine that comes from the WorksOnArm initiative. I applied for it on behalf of the QEMU project and we can have it for as long as it's useful.

I don't know if we need anything more that documenting the nominal contacts in:

Quoted 9 lines
Quoted 2 lines

https://wiki.qemu.org/AdminContacts

That's enough indeed, thanks. I'll follow up with a proposal about the expected duties of admins, which should be nothing but common sense.

Is there anyone that can speak for the s390x machine?

I can be the backup here, if needed.

Willian

#39 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Philippe Mathieu-Daudé
Quoted 5 lines · Philippe Mathieu-Daudé

On 6/8/21 5:14 AM, Cleber Rosa wrote:
Sponsors usually expect some advertising to thanks them, and often regular reports on how their resources are used, else they might not renew their offer. Who should care to keep the relationship with sponsors?

Most sponsors I've encountered did not ask for advertising. Either they have an interest in test coverage because they ship QEMU to customers or they provide resources to open source projects and leave all the detail up to us (they don't need test reports).

But in any case it's easy to have a page on the wiki or website listing sponsors.

There needs to be a point of contact. Someone who answers questions and coordinates with sponsors.

Stefan

#40 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa

Hi,

Quoted 20 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
  1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +3 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
  # guarantees a fresh repository on each job run.
  variables:
    GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task

.gitlab-ci.d/custom-runners.yml line 27

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'

Should it restrict the job for pushes to qemu-project only? If yes, then it probably needs the statement:

'$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'

If you change that here, you will end it changing all over the jobs. In general, there are many boilerplates in this file. I'm ok to merge it as is as long as it is followed by another series to refactor the code.

Quoted 17 lines
.gitlab-ci.d/custom-runners.yml +17 -0
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x

.gitlab-ci.d/custom-runners.yml line 54

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:

Maybe we don't need both ubuntu-18.04-s390x-all and ubuntu-18.04-s390x-alldbg jobs.

Quoted 36 lines
.gitlab-ci.d/custom-runners.yml +36 -0
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build

.gitlab-ci.d/custom-runners.yml line 101

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`

I think it needs to make check-tcg at least. See "build-tci" in .gitlab-ci.d/buildtest.yml for other tests being executed on shared runners.

.gitlab-ci.d/custom-runners.yml line 102

Quoted 2 lines
.gitlab-ci.d/custom-runners.yml +2 -0
+
+ubuntu-18.04-s390x-notcg:

The "build-tcg-disabled" in .gitlab-ci.d/buildtest.yml could be mimic-ed here too.

Quoted 119 lines
.gitlab-ci.d/custom-runners.yml +119 -0
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
#41 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 114 lines · Wainer dos Santos Moschetta

Hi,

Quoted 20 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
  1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +3 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
  # guarantees a fresh repository on each job run.
  variables:
    GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task

.gitlab-ci.d/custom-runners.yml line 27

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'

Should it restrict the job for pushes to qemu-project only? If yes, then it probably needs the statement:

'$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'

If you change that here, you will end it changing all over the jobs. In general, there are many boilerplates in this file. I'm ok to merge it as is as long as it is followed by another series to refactor the code.

Quoted 17 lines
.gitlab-ci.d/custom-runners.yml +17 -0
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x

.gitlab-ci.d/custom-runners.yml line 54

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:

Maybe we don't need both ubuntu-18.04-s390x-all and ubuntu-18.04-s390x-alldbg jobs.

Quoted 41 lines
.gitlab-ci.d/custom-runners.yml +41 -0
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
Quoted 8 lines

.gitlab-ci.d/custom-runners.yml line 101

Quoted 5 lines
.gitlab-ci.d/custom-runners.yml +5 -0
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`

I think it needs to make check-tcg at least. See "build-tci" in .gitlab-ci.d/buildtest.yml for other tests being executed on shared runners.

To get anything other than the s390x-linux-user tests we will need the cross compilers installed. Currently we don't really use docker for anything other than x86_64 hosts (and some aarch64 which I've tested).

Quoted 123 lines

.gitlab-ci.d/custom-runners.yml line 102

Quoted 2 lines
.gitlab-ci.d/custom-runners.yml +2 -0
+
+ubuntu-18.04-s390x-notcg:

The "build-tcg-disabled" in .gitlab-ci.d/buildtest.yml could be mimic-ed here too.

Quoted 119 lines
.gitlab-ci.d/custom-runners.yml +119 -0
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
Signature · 1 line
Alex Bennée
#42 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Wainer dos Santos Moschetta
Quoted 38 lines · Wainer dos Santos Moschetta

Hi,

Quoted 20 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
  1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +3 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
  # guarantees a fresh repository on each job run.
  variables:
    GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task

.gitlab-ci.d/custom-runners.yml line 27

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'

Should it restrict the job for pushes to qemu-project only? If yes, then it probably needs the statement:

Quoted 2 lines

'$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'

I'm not sure we should. In theory, if people have access to other machines on their own accounts, they should be able to trigger the same jobs just by using the "staging" prefix.

Quoted 4 lines

If you change that here, you will end it changing all over the jobs. In general, there are many boilerplates in this file. I'm ok to merge it as is as long as it is followed by another series to refactor the code.

Absolutely, as mentioned before, this is a straightforward mapping of Peter's jobs, so I don't want to go over too many inderaction and abstraction levels initially.

Quoted 78 lines
Quoted 17 lines
.gitlab-ci.d/custom-runners.yml +17 -0
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x

.gitlab-ci.d/custom-runners.yml line 54

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:

Maybe we don't need both ubuntu-18.04-s390x-all and ubuntu-18.04-s390x-alldbg jobs.

Quoted 36 lines
.gitlab-ci.d/custom-runners.yml +36 -0
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build

.gitlab-ci.d/custom-runners.yml line 101

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`

I think it needs to make check-tcg at least. See "build-tci" in .gitlab-ci.d/buildtest.yml for other tests being executed on shared runners.

Quoted 4 lines

.gitlab-ci.d/custom-runners.yml line 102

Quoted 2 lines
.gitlab-ci.d/custom-runners.yml +2 -0
+
+ubuntu-18.04-s390x-notcg:

The "build-tcg-disabled" in .gitlab-ci.d/buildtest.yml could be mimic-ed here too.

These are straightforward mappings of Peter's jobs... I honestly don't think we should deviate any further at this time. Let me know if you think I'm missing something though.

Thanks for the review!
- Cleber.

Reviewed-by on v6 4/4
#43 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa
Quoted 10 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)

Based on the comment from the cover letter that these jobs are defined as trying to mimic what Peter runs on staging, the code looks good to me, so:

#44 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa
Quoted 20 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
  1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +3 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
  # guarantees a fresh repository on each job run.
  variables:
    GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task

.gitlab-ci.d/custom-runners.yml line 27

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'

I don't think this will work very well... sub-maintainers might want to push to a "staging" branch in their forked repositories, and without the s390x runner, the pipeline gets stuck now:

https://gitlab.com/thuth/qemu/-/pipelines/317812558

We had the same issue in the kvm-unit-test CI, and we solved it there by rather making it depend on an environment variable that has to be set if the runner is available:

only:
variables:
- $S390X_RUNNER_AVAILABLE

I think that's also nicer in case someone brings their own s390x runner and want to use the CI tests on other branches than staging.

Could you please change your patch accordingly?

Thanks,
Thomas

#45 lore raw

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 37 lines · Thomas Huth
Quoted 20 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
  1 file changed, 208 insertions(+)
.gitlab-ci.d/custom-runners.yml +3 -0
index a07b27384c..061d3cdfed 100644
@@ -12,3 +12,211 @@
  # guarantees a fresh repository on each job run.
  variables:
    GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task

.gitlab-ci.d/custom-runners.yml line 27

Quoted 10 lines
.gitlab-ci.d/custom-runners.yml +10 -0
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'

I don't think this will work very well... sub-maintainers might want to push to a "staging" branch in their forked repositories, and without the s390x runner, the pipeline gets stuck now:

Quoted 2 lines

https://gitlab.com/thuth/qemu/-/pipelines/317812558

Hi Thomas,

As I put it in another response, I saw that actually as a feature, in the sense that:

* people should indeed be allowed to push to their repos and leverage their hardware, and
* "staging" is a pretty well scoped word, and has a reasonably well defined meaning
* one would want to mimic as closely as possible what will be done before a PR is merged

I agree that having the jobs stuck in any situation is not ideal, but I honestly find that it would be reasonably hard to accidentally hit that situation. I also believe it will end up being inevitable for entities to do a meta-analysis of the GitLab CI pipeline results, possibly disregarding jobs that they can not run, or simply do not care about, in their forks.

Quoted 11 lines

We had the same issue in the kvm-unit-test CI, and we solved it there by rather making it depend on an environment variable that has to be set if the runner is available:

only:
variables:
- $S390X_RUNNER_AVAILABLE

I think that's also nicer in case someone brings their own s390x runner and want to use the CI tests on other branches than staging.

The problem with this approach, is that it would not be enough to protect the jobs based on variables for the architecture, as the OS type and version also play a part in the possibility of running jobs. For instance, suppose we get s390x machines from LinuxOne running RHEL. We'd need variables such as, say, S390X_RHEL_8_4_RUNNER_AVAILABLE and S390X_RHEL_7_6_RUNNER_AVAILABLE.

Quoted 2 lines

Could you please change your patch accordingly?

If you strongly believe now is the time to attempt to handle that problem, I can go ahead and change it. I stand behind my original position that we should start with a simpler, "by convention" approach and address the more complex scenarios as/if they come up.

Quoted 3 lines

Thanks,
Thomas

Thank you!
- Cleber.