Skip to contentSkip to conversation

GitLab Custom Runners and Jobs (was: QEMU Gating CI)

PATCH Superseded Cleber Rosa 6 participants 29 messages Oct 19, 2020 – Feb 4, 2021

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover GitLab Custom Runners and Jobs (was: QEMU Gating CI)

TL;DR: this should allow the QEMU maintainer to push to the staging branch, and have custom jobs running on the project's aarch64 and s390x machines. Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

At this point, there are two specific runners (an aarch64 and an s390) registered with GitLab, at https://gitlab.com/qemu-project, currently setup to the "qemu" repository.

Changes from v3:

- Applied changes to match <[email protected]>, that is, added ninja-build to "build-environment.yml" list of packages and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners", given that the other jobs running on shared runners are also considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
* update -> up to date (Erik)
* a different set of CI jobs -> different CI jobs (Erik)
* Pull requests will only be merged -> code will only be merged (Stefan)
* Setup -> set up (Stefan)
* them -> they (Stefan)
* the -> where the (Stefan)
* dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that no external variable is used there

- Updated package list in "build-environment.yml" from current dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30), and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts" into (Phillipe):
1) Basic documentation and configuration (gitlab-ci.yml) placeholder
2) Playbooks for setting up a build environment
3) Playbooks for setting up gitlab-runner
4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine build and test.

- Set custom jobs to not depend on any other job, so they can start right away.

- Set rules for starting jobs so that all pushing to any branch that start with name "staging". This allows the project maintainer to use the "push to staging" workflow, while also allowing others to generate similar jobs. If this project has configured custom runners, the jobs will run, if not, the pipeline will be marked as "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in ".gitlab-ci.yml", assumes that the additional jobs on the staging branch running on the freely available gitlab shared runner are positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to simply 18.04 (assumes no diverse minor levels will be used or specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
Jobs based on custom runners: documentation and configuration
placeholder
Jobs based on custom runners: build environment docs and playbook Jobs based on custom runners: docs and gitlab-runner setup playbook Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 206 ++++++++++++++++++++++
 .gitlab-ci.yml                         |   1 +
 docs/devel/ci.rst                      | 149 ++++++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   1 +
 scripts/ci/setup/build-environment.yml | 233 +++++++++++++++++++++++++
 scripts/ci/setup/gitlab-runner.yml     |  72 ++++++++
 scripts/ci/setup/inventory             |   2 +
 scripts/ci/setup/vars.yml.template     |  13 ++
 9 files changed, 678 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory
 create mode 100644 scripts/ci/setup/vars.yml.template
Signature · 1 line
2.25.4
#1 lore raw

PATCH v4 1/4 Jobs based on custom runners: documentation and configuration placeholder

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place.

Those jobs are intended to run on hardware and/or Operating Systems not provided by GitLab's shared runners.

---
 .gitlab-ci.d/custom-runners.yml | 14 +++++++++
 .gitlab-ci.yml                  |  1 +
 docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 70 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
.gitlab-ci.d/custom-runners.yml new file +14 -0
new file mode 100644
index 0000000000..3004da2bda
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the recursive submodule
+# strategy.
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
.gitlab-ci.yml +1 -0
index 8ffd415ca5..b33c433fd7 100644
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
docs/devel/ci.rst new file +54 -0

54 lines not on this page · raw message

docs/devel/index.rst +1 -0
index 77baae5c77..2fdd36e751 100644
@@ -21,6 +21,7 @@ Contents:
    atomics
    stable-process
    testing
+   ci
    qtest
    decodetree
    secure-coding-practices
Signature · 1 line
2.25.4
#2 lore raw

Re: [PATCH v4 1/4] Jobs based on custom runners: documentation and configuration placeholder

Quoted 69 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place.

Those jobs are intended to run on hardware and/or Operating Systems not provided by GitLab's shared runners.

---
 .gitlab-ci.d/custom-runners.yml | 14 +++++++++
 .gitlab-ci.yml                  |  1 +
 docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 70 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
.gitlab-ci.d/custom-runners.yml new file +14 -0
new file mode 100644
index 0000000000..3004da2bda
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the recursive submodule
+# strategy.
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
.gitlab-ci.yml +1 -0
index 8ffd415ca5..b33c433fd7 100644
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
docs/devel/ci.rst new file +14 -0
new file mode 100644
index 0000000000..41a4bbddad
@@ -0,0 +1,54 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there

docs/devel/ci.rst line 24

Quoted 10 lines
docs/devel/ci.rst new file +10 -0
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner" here.

Nit: Remove one of the two "here" in the above sentence.

docs/devel/ci.rst line 37

Quoted 12 lines
docs/devel/ci.rst new file +12 -0
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Current Jobs
+------------
+
+The current CI jobs based on custom runners have the primary goal of
+catching and preventing regressions on a wider number of host systems
+than the ones provided by GitLab's shared runners.
+
+Also, the mechanics of reliability, capacity and overall maintanance

s/maintanance/maintenance/

docs/devel/ci.rst line 39

Quoted 2 lines
docs/devel/ci.rst new file +2 -0
+of the machines provided by the QEMU project itself for those jobs
+will be evaluated.

I'm not sure what this sentence is really good for... of course new stuff has to prove its usefulness first, but that's always the case and does not need to be mentioned in the documentation, I think? ... maybe that sentence is better something for the patch description instead of (hopefully) long-lasting documentation here?

Quoted 20 lines
docs/devel/ci.rst new file +14 -0
+Future Plans and Jobs
+---------------------
+
+Once the CI Jobs based on custom runners have been proved mature with
+the initial set of jobs run on machines from the QEMU project, other
+members in the community should be able provide their own machine
+configuration documentation/scripts, and accompanying job definitions.
+
+As a general rule, those newly added contributed jobs should run as
+"non-gating", until their reliability is verified.
+
+The precise minimum requirements and exact rules for machine
+configuration documentation/scripts, and the success rate of jobs are
+still to be defined.
docs/devel/index.rst +0 -0
index 77baae5c77..2fdd36e751 100644

docs/devel/index.rst Contents: · line 27

Quoted 7 lines
docs/devel/index.rst +1 -0
@@ -21,6 +21,7 @@ Contents:
    atomics
    stable-process
    testing
+   ci
    qtest
    decodetree
    secure-coding-practices

Thomas

#3 lore raw

Re: [PATCH v4 1/4] Jobs based on custom runners: documentation and configuration placeholder

Quoted 83 lines · Thomas Huth
Quoted 69 lines · Cleber Rosa

As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place.

Those jobs are intended to run on hardware and/or Operating Systems not provided by GitLab's shared runners.

---
 .gitlab-ci.d/custom-runners.yml | 14 +++++++++
 .gitlab-ci.yml                  |  1 +
 docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 70 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
.gitlab-ci.d/custom-runners.yml new file +14 -0
new file mode 100644
index 0000000000..3004da2bda
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the recursive submodule
+# strategy.
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
.gitlab-ci.yml +1 -0
index 8ffd415ca5..b33c433fd7 100644
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
docs/devel/ci.rst new file +14 -0
new file mode 100644
index 0000000000..41a4bbddad
@@ -0,0 +1,54 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there

docs/devel/ci.rst line 24

Quoted 10 lines
docs/devel/ci.rst new file +10 -0
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner" here.

Nit: Remove one of the two "here" in the above sentence.

Quoted 15 lines

docs/devel/ci.rst line 37

Quoted 12 lines
docs/devel/ci.rst new file +12 -0
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Current Jobs
+------------
+
+The current CI jobs based on custom runners have the primary goal of
+catching and preventing regressions on a wider number of host systems
+than the ones provided by GitLab's shared runners.
+
+Also, the mechanics of reliability, capacity and overall maintanance

s/maintanance/maintenance/

Oopsie... thanks.

Quoted 9 lines

docs/devel/ci.rst line 39

Quoted 2 lines
docs/devel/ci.rst new file +2 -0
+of the machines provided by the QEMU project itself for those jobs
+will be evaluated.

I'm not sure what this sentence is really good for... of course new stuff has to prove its usefulness first, but that's always the case and does not need to be mentioned in the documentation, I think? ... maybe that sentence is better something for the patch description instead of (hopefully) long-lasting documentation here?

I think the statement attempts to set the tone here, and answer a question that I've seen more than once... It certainly feels a bit out of place here, but I also think the commit message would be too transient and almost invisible. I expect a lot will change in the CI and in its docs, so I don't think this exact sentence will be too long lasting.

Let me know if that convinces you it deserves to be in the docs, if not I'll gladly moved it to the commit message.

Thanks!
- Cleber

#4 lore raw

PATCH v4 2/4 Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---
 docs/devel/ci.rst                      |  32 ++++
 scripts/ci/setup/build-environment.yml | 233 +++++++++++++++++++++++++
 scripts/ci/setup/inventory             |   2 +
 3 files changed, 267 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory
docs/devel/ci.rst +32 -0
index 41a4bbddad..208b5e399b 100644
@@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as
 The precise minimum requirements and exact rules for machine
 configuration documentation/scripts, and the success rate of jobs are
 still to be defined.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Start by adding your machines to
+the ``inventory`` file under ``scripts/ci/setup``, such as this::
+
+  [local]
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  [local]
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
scripts/ci/setup/build-environment.yml new file +233 -0

233 lines not on this page · raw message

scripts/ci/setup/inventory new file +2 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]
+localhost
Signature · 1 line
2.25.4
#5 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

...

scripts/ci/setup/inventory line 1

Quoted 7 lines
scripts/ci/setup/inventory new file +1 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]

Nit pick, is a group for localhost actually needed?

Regards,
Erik

#6 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 12 lines · Erik Skultety

...

scripts/ci/setup/inventory line 1

Quoted 7 lines
scripts/ci/setup/inventory new file +1 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]

Nit pick, is a group for localhost actually needed?

You're right, it's not needed... I just thought it gave the "localhost" entry some "shelter" and "context". :)

And, I think a mostly "ini-like" file without a section triggers an OCD reaction in me. I can remove it if it does something similar to you! :)

Thanks!
- Cleber.

Quoted 2 lines

Regards,
Erik

#7 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 14 lines · Cleber Rosa
Quoted 12 lines · Erik Skultety

...

scripts/ci/setup/inventory line 1

Quoted 7 lines
scripts/ci/setup/inventory new file +1 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]

Nit pick, is a group for localhost actually needed?

Quoted 8 lines

You're right, it's not needed... I just thought it gave the "localhost" entry some "shelter" and "context". :)

And, I think a mostly "ini-like" file without a section triggers an OCD reaction in me. I can remove it if it does something similar to you! :)

Thanks!
- Cleber.

I understand, but even though it may be an ini-like config, "sections" have a very distinct meaning in Ansible and that is grouping, so unless there are hosts to group, I'd keep the inventory in a form of a simple list of individual hosts.

Regards,
Erik

#8 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 21 lines · Erik Skultety
Quoted 20 lines · Cleber Rosa
Quoted 12 lines · Erik Skultety

...

scripts/ci/setup/inventory line 1

Quoted 7 lines
scripts/ci/setup/inventory new file +1 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]

Nit pick, is a group for localhost actually needed?

You're right, it's not needed... I just thought it gave the "localhost" entry some "shelter" and "context". :)

And, I think a mostly "ini-like" file without a section triggers an OCD reaction in me. I can remove it if it does something similar to you! :)

Quoted 8 lines
Quoted 2 lines

Thanks!
- Cleber.

I understand, but even though it may be an ini-like config, "sections" have a very distinct meaning in Ansible and that is grouping, so unless there are hosts to group, I'd keep the inventory in a form of a simple list of individual hosts.

Fair enough. Changing it for v5.

Thanks!
- Cleber.

Quoted 2 lines

Regards,
Erik

#9 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa

scripts/ci/setup/build-environment.yml line 4

Quoted 6 lines · Cleber Rosa

+++ b/scripts/ci/setup/build-environment.yml

scripts/ci/setup/build-environment.yml new file +4 -0
@@ -0,0 +1,233 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:

My Ansible-fu is a bit rusty at the moment, so please double-check my claims and apologies in advance for the ones that I will get wrong O:-)

scripts/ci/setup/build-environment.yml line 11

Quoted 7 lines
scripts/ci/setup/build-environment.yml new file +7 -0
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      apt:
+        update_cache: yes
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+        pkg:
+          - ccache
+          - clang

Instead of using the 'apt' module here, and the equivalent module further down, you could just do

package:
name:
- pkg1
- pkg2
...
state: present

every single time and let Ansible take care of the differences for you.

scripts/ci/setup/build-environment.yml line 60

Quoted 1 line
scripts/ci/setup/build-environment.yml new file +1 -0
+      when: "ansible_facts['distribution'] == 'Ubuntu'"

Quoting the entire condition is not necessary, you can just have

when: ansible_facts['distribution'] == 'Ubuntu'

or, my preference,

when:
- ansible_facts['distribution'] == 'Ubuntu'

which results in a nicer diff when you add/remove conditions.

scripts/ci/setup/build-environment.yml line 67

Quoted 6 lines
scripts/ci/setup/build-environment.yml new file +6 -0
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      apt:
+        update_cache: yes
+        pkg:
+         - libspice-server-dev
+         - libxen-dev

Indentation of list items is inconsistent here.

scripts/ci/setup/build-environment.yml line 76

Quoted 4 lines
scripts/ci/setup/build-environment.yml new file +4 -0
+    - name: Install basic packages to build QEMU on FreeBSD 12.x
+      pkgng:
+        # Originally from packages on .cirrus.yml under the freebsd_12_task
+        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir

See above for 'pkgng' vs 'package', but at the very least this should be

pkgng:
name:
- bash
- curl
...

or each time the list is touched the resulting diff is going to be unmanageable.

scripts/ci/setup/build-environment.yml line 198

Quoted 9 lines
scripts/ci/setup/build-environment.yml new file +9 -0
+    - name: Enable PowerTools repo on CentOS 8
+      ini_file:
+        path: /etc/yum.repos.d/CentOS-PowerTools.repo
+        section: PowerTools
+        option: enabled
+        value: "1"
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"

Another option would be to use

  command: 'dnf config-manager --set-enabled Stream-PowerTools -y'
  args:
    warn: no

but I have to admit the way you're doing it is very clever ;)

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#10 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 29 lines · Andrea Bolognani

scripts/ci/setup/build-environment.yml line 4

Quoted 6 lines · Cleber Rosa

+++ b/scripts/ci/setup/build-environment.yml

scripts/ci/setup/build-environment.yml new file +4 -0
@@ -0,0 +1,233 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:

My Ansible-fu is a bit rusty at the moment, so please double-check my claims and apologies in advance for the ones that I will get wrong O:-)

scripts/ci/setup/build-environment.yml line 11

Quoted 7 lines
scripts/ci/setup/build-environment.yml new file +7 -0
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      apt:
+        update_cache: yes
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+        pkg:
+          - ccache
+          - clang

Instead of using the 'apt' module here, and the equivalent module further down, you could just do

package:
name:
- pkg1
- pkg2
...
state: present

Quoted 3 lines

every single time and let Ansible take care of the differences for you.

I'm almost sure that this was a conscious decision. I remeber it had to do with not being able to set update_cache, and failures on recently installed systems and containers that did not update the APT cache. There may be something else, but I'll have to give it another round of testing.

FIY, under the hood, package is not really a module, but an action plugin that forwards these very limited options to the set or detected package manager, so it brings uniformity in the playbook, but limits the control too. IMO, it's very low impact to leave it AS IS.

Quoted 11 lines

scripts/ci/setup/build-environment.yml line 60

Quoted 1 line
scripts/ci/setup/build-environment.yml new file +1 -0
+      when: "ansible_facts['distribution'] == 'Ubuntu'"

Quoting the entire condition is not necessary, you can just have

when: ansible_facts['distribution'] == 'Ubuntu'

or, my preference,

when:
- ansible_facts['distribution'] == 'Ubuntu'

Yep, I've used the explicit lists when there was more than one condition, but having a standard style is better indeed.

Quoted 11 lines

which results in a nicer diff when you add/remove conditions.

scripts/ci/setup/build-environment.yml line 67

Quoted 6 lines
scripts/ci/setup/build-environment.yml new file +6 -0
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      apt:
+        update_cache: yes
+        pkg:
+         - libspice-server-dev
+         - libxen-dev

Indentation of list items is inconsistent here.

True. Fixed, thanks!

Quoted 8 lines

scripts/ci/setup/build-environment.yml line 76

Quoted 4 lines
scripts/ci/setup/build-environment.yml new file +4 -0
+    - name: Install basic packages to build QEMU on FreeBSD 12.x
+      pkgng:
+        # Originally from packages on .cirrus.yml under the freebsd_12_task
+        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir

See above for 'pkgng' vs 'package', but at the very least this should be

Quoted 9 lines

pkgng:
name:
- bash
- curl
...

or each time the list is touched the resulting diff is going to be unmanageable.

The documentation suggests a comma separated list of package names:

https://docs.ansible.com/ansible/2.8/modules/pkgng_module.html#pkgng-module

And the reason is that this module is not as smart as others, and will run one separate command for each individual package name value:

https://github.com/ansible/ansible/blob/v2.10.3/test/support/integration/plugins/modules/pkgng.py#L214

It's a tradeoff indeed, but I think we're aligned with the docs.

Quoted 10 lines

scripts/ci/setup/build-environment.yml line 198

Quoted 9 lines
scripts/ci/setup/build-environment.yml new file +9 -0
+    - name: Enable PowerTools repo on CentOS 8
+      ini_file:
+        path: /etc/yum.repos.d/CentOS-PowerTools.repo
+        section: PowerTools
+        option: enabled
+        value: "1"
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
Quoted 8 lines

Another option would be to use

  command: 'dnf config-manager --set-enabled Stream-PowerTools -y'
  args:
    warn: no

but I have to admit the way you're doing it is very clever ;)

Yeah, that would require another package to be installed, and then the command executed... So I think this is cheaper and eaiser indeed.

Quoted 3 lines
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization

Thanks for the review, I'll report on the additional points as soon as I test them. If appropriate, I'll put notes on the v5.

- Cleber.

#11 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 21 lines · Cleber Rosa
Quoted 9 lines · Andrea Bolognani

scripts/ci/setup/build-environment.yml line 76

Quoted 4 lines · Cleber Rosa
scripts/ci/setup/build-environment.yml new file +4 -0
+    - name: Install basic packages to build QEMU on FreeBSD 12.x
+      pkgng:
+        # Originally from packages on .cirrus.yml under the freebsd_12_task
+        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir

See above for 'pkgng' vs 'package', but at the very least this should be

Quoted 8 lines

pkgng:
name:
- bash
- curl
...

or each time the list is touched the resulting diff is going to be unmanageable.

The documentation suggests a comma separated list of package names:

Quoted 8 lines

https://docs.ansible.com/ansible/2.8/modules/pkgng_module.html#pkgng-module

And the reason is that this module is not as smart as others, and will run one separate command for each individual package name value:

https://github.com/ansible/ansible/blob/v2.10.3/test/support/integration/plugins/modules/pkgng.py#L214

It's a tradeoff indeed, but I think we're aligned with the docs.

I would probably take the performance hit over having to deal with an unmaintainable blob. As a rule of thumb, reviewer time is more valuable than machine time.

If the performance hit turns out to be unacceptably big, you could at least do something along the lines of

- set_fact:
freebsd_packages:
- bash
- curl
...

- pkgng:

      name: "{ freebsd_packages | join(',') }"

to keep things reviewable.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#12 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

In reply to Cleber Rosa
Quoted 10 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---

[...]

Quoted 5 lines
scripts/ci/setup/build-environment.yml new file +5 -0
+    - name: Enable PowerTools repo on CentOS 8
+      ini_file:
+        path: /etc/yum.repos.d/CentOS-PowerTools.repo
+        section: PowerTools
+        option: enabled

scripts/ci/setup/build-environment.yml line 204

Quoted 10 lines
scripts/ci/setup/build-environment.yml new file +10 -0
+        value: "1"
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
+
+    - name: Install basic packages to build QEMU on CentOS 8
+      dnf:
+        # Originally from tests/docker/dockerfiles/centos8.docker
+        name:
+          - SDL-devel

We do not support SDL1 in QEMU anymore, so this should be SDL2-devel now. Yes, we've also got it wrong in the docker files ... I'll send a patch to fix it there.

Thomas

#13 lore raw

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

Quoted 28 lines · Thomas Huth
Quoted 10 lines · Cleber Rosa

To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---

[...]

Quoted 5 lines
scripts/ci/setup/build-environment.yml new file +5 -0
+    - name: Enable PowerTools repo on CentOS 8
+      ini_file:
+        path: /etc/yum.repos.d/CentOS-PowerTools.repo
+        section: PowerTools
+        option: enabled

scripts/ci/setup/build-environment.yml line 204

Quoted 10 lines
scripts/ci/setup/build-environment.yml new file +10 -0
+        value: "1"
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
+
+    - name: Install basic packages to build QEMU on CentOS 8
+      dnf:
+        # Originally from tests/docker/dockerfiles/centos8.docker
+        name:
+          - SDL-devel
Quoted 5 lines

We do not support SDL1 in QEMU anymore, so this should be SDL2-devel now. Yes, we've also got it wrong in the docker files ... I'll send a patch to fix it there.

Thomas

Nice catch. I see the dockerfile changes are already merged, so updating them here.

- Cleber.

#14 lore raw

PATCH v4 3/4 Jobs based on custom runners: docs and gitlab-runner setup playbook

In reply to Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---
 docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 149 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +63 -0
index 208b5e399b..a234a5e24c 100644
@@ -84,3 +84,66 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at
+          https://docs.gitlab.com/runner/install/freebsd.html
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+FreeBSD 12.1 x86_64 system, the tags should be set as::
+
+  freebsd12.1,x86_64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  freebsd-12.1-x86_64-all:
+   tags:
+   - freebsd_12.1
+   - x86_64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * uncheck the "Run untagged jobs" check box
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +72 -0

72 lines not on this page · raw message

scripts/ci/setup/vars.yml.template new file +13 -0
new file mode 100644
index 0000000000..621435d030
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.1.1
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
Signature · 1 line
2.25.4
#15 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 5 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

Quoted 7 lines

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---

In general, there's been put quite some effort into the playbooks - sorry I'm late to the game - is there a plan to introduce QEMU as a project to lcitool? We've taken care of most of the bits in the playbooks that are being introduced and for the remaining ones I think it would be that big of an overhaul to do the adjustments. One major re-factor though would IMO be to break the dependency lcitool has on the machine naming, kind of restricting it to a limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which makes it inconvenient to prepare physical hosts.

More comments inline...

Quoted 32 lines
 docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 149 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +15 -0
index 208b5e399b..a234a5e24c 100644
@@ -84,3 +84,66 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+

docs/devel/ci.rst To run the playbook, execute:: · line 111

Quoted 10 lines
docs/devel/ci.rst +10 -0
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``

Yes, this can be suboptimal...Would it make sense to fall back to build the binary of a given version from git as a fallback during this playbook if the necessary arch version isn't provided the official way? Just an idea, I'd like to avoid the need for you to become the maintainer of the binaries and keep up with the releases.

docs/devel/ci.rst To run the playbook, execute:: · line 120

Quoted 9 lines
docs/devel/ci.rst +9 -0
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at

Which one of them is considered an automation problem? In lcitool we made gitlab-runner completely automated on all distros, including FreeBSD:

4) log file permissions - you're creating the user, you can as well create the file with correct permissions

5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml it will pull a bunch of dependencies which ensure the dir exists

6) gitlab_runner service script should IMO provided by this automation as a template and install to the correct location

7) ensure the service script is executable - template module can do that

8) register the runner - you're doing that

9) enable the service - Ansible's service module is generic and supports init/OpenRC

10) I don't see a step 10

IOW I think there should be as little manual intervention as possible and in this case I don't think any manual steps are needed by the user.

Regards,
Erik

Quoted 50 lines
docs/devel/ci.rst +29 -0
+          https://docs.gitlab.com/runner/install/freebsd.html
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+FreeBSD 12.1 x86_64 system, the tags should be set as::
+
+  freebsd12.1,x86_64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  freebsd-12.1-x86_64-all:
+   tags:
+   - freebsd_12.1
+   - x86_64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * uncheck the "Run untagged jobs" check box
+ * give it a better Description
scripts/ci/setup/.gitignore new file +1 -0
new file mode 100644
index 0000000000..f112d05dd0
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
scripts/ci/setup/gitlab-runner.yml new file +7 -0
new file mode 100644
index 0000000000..c2f52dad10
@@ -0,0 +1,72 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:

scripts/ci/setup/gitlab-runner.yml line 17

Quoted 10 lines
scripts/ci/setup/gitlab-runner.yml new file +10 -0
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403

Why is 403 an acceptable success status code?

Quoted 77 lines
scripts/ci/setup/gitlab-runner.yml new file +55 -0
+      register: gitlab_runner_available_archive
+
+    - name: Update base url
+      set_fact:
+        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
+      when: gitlab_runner_available_archive.status == 200
+    - debug:
+        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
+
+    - name: Set OS name (FreeBSD)
+      set_fact:
+        gitlab_runner_os: freebsd
+      when: "ansible_facts['system'] == 'FreeBSD'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Downloads the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
scripts/ci/setup/vars.yml.template new file +13 -0
new file mode 100644
index 0000000000..621435d030
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.1.1
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
Signature · 1 line
2.25.4
#16 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 10 lines · Erik Skultety
Quoted 9 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

Quoted 6 lines
Quoted 3 lines

---

In general, there's been put quite some effort into the playbooks - sorry I'm late to the game - is there a plan to introduce QEMU as a project to lcitool?

I think it's becoming quite clear that having so much duplication (in the dockerfiles, tests/vm, this playbook, etc) is costly and error prone. I don't know if anyone has invested time in a PoC to consolidate those (with lcitool), but I can certainly see the upside to that. BTW, are you volunteering (wink wink)? :)

Quoted 7 lines

We've taken care of most of the bits in the playbooks that are being introduced and for the remaining ones I think it would be that big of an overhaul to do the adjustments. One major re-factor though would IMO be to break the dependency lcitool has on the machine naming, kind of restricting it to a limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which makes it inconvenient to prepare physical hosts.

Right... I wasn't aware of that depedency. And, this may be a nice project to make sure that lcitool doesn't have any other libvirt specificities.

Quoted 42 lines

More comments inline...

Quoted 40 lines
 docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 149 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +23 -0
index 208b5e399b..a234a5e24c 100644
@@ -84,3 +84,66 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
Quoted 9 lines

docs/devel/ci.rst To run the playbook, execute:: · line 111

Quoted 2 lines
docs/devel/ci.rst +2 -0
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``

Yes, this can be suboptimal...Would it make sense to fall back to build the binary of a given version from git as a fallback during this playbook if the necessary arch version isn't provided the official way? Just an idea, I'd like to avoid the need for you to become the maintainer of the binaries and keep up with the releases.

Well, building them during the playbook would be a lot more complex... You can have your own repo with your own builds, and just tweak your vars.yml.

Quoted 4 lines
Quoted 4 lines
docs/devel/ci.rst +4 -0
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
Quoted 9 lines

docs/devel/ci.rst To run the playbook, execute:: · line 120

Quoted 5 lines
docs/devel/ci.rst +5 -0
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at

Which one of them is considered an automation problem? In lcitool we made gitlab-runner completely automated on all distros, including FreeBSD:

It looks like lcitool went the more practical route. I was hoping to not have to treat gitlab-runner in such a special way in any "supported" OS. What I mean is, I'd rather write the code within gitlab-runner (or reespective libraries). Of course, I did not get to it, so that's why I just documented the steps here.

I'll take a look at lcitool's playbook to see if I can easily canibalize some of that. But, at this time, we don't runners for FreeBSD anyway, so I guess this is not *that* urgent.

Quoted 9 lines

4) log file permissions - you're creating the user, you can as well create the file with correct permissions

5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml it will pull a bunch of dependencies which ensure the dir exists

6) gitlab_runner service script should IMO provided by this automation as a template and install to the correct location

Quoted 9 lines

7) ensure the service script is executable - template module can do that

8) register the runner - you're doing that

9) enable the service - Ansible's service module is generic and supports init/OpenRC

10) I don't see a step 10

This was either a mistake or the installation steps got updated. I'll adjust that.

Quoted 3 lines

IOW I think there should be as little manual intervention as possible and in this case I don't think any manual steps are needed by the user.

Agreed. I was not super happy with the current state of this playbook wrt FreeBSD, but I decided to pick other battles to fight. Also, newer gitlab-runner versions *may* be using the updated service management lib, which may reduce/remove the need for custom handling.

Anyway, I'll look at what can be improved here, considering the cost.

Thanks!
- Cleber.

#17 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 15 lines · Cleber Rosa
Quoted 14 lines · Erik Skultety
Quoted 12 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---

Quoted 8 lines
Quoted 2 lines

In general, there's been put quite some effort into the playbooks - sorry I'm late to the game - is there a plan to introduce QEMU as a project to lcitool?

I think it's becoming quite clear that having so much duplication (in the dockerfiles, tests/vm, this playbook, etc) is costly and error prone. I don't know if anyone has invested time in a PoC to consolidate those (with lcitool), but I can certainly see the upside to that. BTW, are you volunteering (wink wink)? :)

I don't think I was trying to :), but sure, I can dedicate some time to it. I'll need a bit of guidance in the QEMU world though for sure.

Quoted 9 lines
Quoted 8 lines

We've taken care of most of the bits in the playbooks that are being introduced and for the remaining ones I think it would be that big of an overhaul to do the adjustments. One major re-factor though would IMO be to break the dependency lcitool has on the machine naming, kind of restricting it to a limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which makes it inconvenient to prepare physical hosts.

Quoted 3 lines

Right... I wasn't aware of that depedency. And, this may be a nice project to make sure that lcitool doesn't have any other libvirt specificities.

Yeah, that one will be a major change. Thinking about this some more, while having a QEMU project inside lcitool sounds definitely useful, it would probably be a wasted effort wrt task priority right now, because without expanding lcitool's range of applicable hosts I don't see how QEMU would utilize the tool without using quirks.

Quoted 53 lines
Quoted 43 lines

More comments inline...

Quoted 40 lines
 docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 149 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template
docs/devel/ci.rst +23 -0
index 208b5e399b..a234a5e24c 100644
@@ -84,3 +84,66 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
Quoted 9 lines

docs/devel/ci.rst To run the playbook, execute:: · line 111

Quoted 2 lines
docs/devel/ci.rst +2 -0
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``

Yes, this can be suboptimal...Would it make sense to fall back to build the binary of a given version from git as a fallback during this playbook if the necessary arch version isn't provided the official way? Just an idea, I'd like to avoid the need for you to become the maintainer of the binaries and keep up with the releases.

Quoted 3 lines

Well, building them during the playbook would be a lot more complex... You can have your own repo with your own builds, and just tweak your vars.yml.

I didn't want to suggest this as a replacement to vars.yml.
What if people wanting to run the playbook don't have their own repo and you (I mean you personally) are not providing a fresh build of gitlab-runner for the given architecture?
The automation machinery would surely be more complex, but at the same time, I can image a dedicated playbook that would handle gitlab-runner building and installation of the binary to the correct place, you'd then simply include that playbook if vars.yml don't specify an alternative repo and download from AWS failed.
Another thing to consider - you currently host the missing arch builds under your personal namespace on fedorapeople, what if you decide one day you're not interested in being involved in QEMU CI that much and suddenly you stop building the binaries, someone has to take over. On the other hand, if you automate this whole process, having exposed the builds would definitely not slow down machine provisions, but would also not cause them to fail if nobody exposes them.

Quoted 21 lines
Quoted 14 lines

docs/devel/ci.rst To run the playbook, execute:: · line 120

Quoted 10 lines
docs/devel/ci.rst +9 -0
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at

Which one of them is considered an automation problem? In lcitool we made gitlab-runner completely automated on all distros, including FreeBSD:

It looks like lcitool went the more practical route. I was hoping to not have to treat gitlab-runner in such a special way in any "supported" OS. What I mean is, I'd rather write the code within gitlab-runner (or reespective libraries). Of course, I did not get to it, so that's why I just documented the steps here.

Quoted 3 lines

I'll take a look at lcitool's playbook to see if I can easily canibalize some of that. But, at this time, we don't runners for FreeBSD anyway, so I guess this is not *that* urgent.

Okay, I was simply curious. If it's not something you don't need at the moment, then let's not focus on it.

Quoted 23 lines
Quoted 10 lines

4) log file permissions - you're creating the user, you can as well create the file with correct permissions

5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml it will pull a bunch of dependencies which ensure the dir exists

6) gitlab_runner service script should IMO provided by this automation as a template and install to the correct location

Quoted 9 lines

7) ensure the service script is executable - template module can do that

8) register the runner - you're doing that

9) enable the service - Ansible's service module is generic and supports init/OpenRC

10) I don't see a step 10

This was either a mistake or the installation steps got updated. I'll adjust that.

Quoted 8 lines
Quoted 3 lines

IOW I think there should be as little manual intervention as possible and in this case I don't think any manual steps are needed by the user.

Agreed. I was not super happy with the current state of this playbook wrt FreeBSD, but I decided to pick other battles to fight. Also, newer gitlab-runner versions *may* be using the updated service management lib, which may reduce/remove the need for custom handling.

What do you mean by "service management lib"?

Regards,
Erik

#18 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 19 lines · Erik Skultety
Quoted 18 lines · Cleber Rosa
Quoted 16 lines · Erik Skultety
Quoted 5 lines · Cleber Rosa

To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps.

Quoted 7 lines

The playbook introduced here covers a number of different Linux distributions and FreeBSD, and are intended to provide a reproducible environment.

---

In general, there's been put quite some effort into the playbooks - sorry I'm late to the game - is there a plan to introduce QEMU as a project to lcitool?

Quoted 8 lines
Quoted 5 lines

I think it's becoming quite clear that having so much duplication (in the dockerfiles, tests/vm, this playbook, etc) is costly and error prone. I don't know if anyone has invested time in a PoC to consolidate those (with lcitool), but I can certainly see the upside to that. BTW, are you volunteering (wink wink)? :)

I don't think I was trying to :), but sure, I can dedicate some time to it. I'll need a bit of guidance in the QEMU world though for sure.

I think the obvious and easy place is start using lcitool is for the tests/docker/dockerfiles/*. All that's required is to add mappings to lcitool for the various deps that QEMU has which libvirt does not already have. Then we should be able to start auto-generating the dockerfiles without too much difficulty. This will be a significant step forward because it will help us keep te package lists in sync across all the dockerfiles which is a major fail in QEMU right now.

Dealing with tests/vm replacement or these ansible recipes is likely to be a significantly more challenging proposition. Perhaps we can again start by just automating creation of the package lists that the tests/vm and ansibile recipes need, as again those are all inconsistent.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#19 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 7 lines · Daniel P. Berrangé

I think the obvious and easy place is start using lcitool is for the tests/docker/dockerfiles/*. All that's required is to add mappings to lcitool for the various deps that QEMU has which libvirt does not already have. Then we should be able to start auto-generating the dockerfiles without too much difficulty. This will be a significant step forward because it will help us keep te package lists in sync across all the dockerfiles which is a major fail in QEMU right now.

I took a quick look at the contents of those dockerfiles and while I agree that it's the obvious place to start, I'm not sure I would say that generating them using lcitool is going to be easy :)

Basically there seem to be a number of assumptions both in lcitool and in the QEMU dockerfiles, and getting the two to agree on those will require quite some work from what I can tell.

But again, I agree it's something that should be worked on.

Quoted 5 lines

Dealing with tests/vm replacement or these ansible recipes is likely to be a significantly more challenging proposition. Perhaps we can again start by just automating creation of the package lists that the tests/vm and ansibile recipes need, as again those are all inconsistent.

Perhaps 'lcitool variables' could be of use here.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#20 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

In reply to Erik Skultety
Quoted 10 lines · Erik Skultety

scripts/ci/setup/gitlab-runner.yml line 17

Quoted 8 lines · Cleber Rosa
scripts/ci/setup/gitlab-runner.yml new file +8 -0
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403
Quoted 2 lines

Why is 403 an acceptable success status code?

I missed this question in my last reply, sorry.

s3 will throw a 403 if the URI doesn't exist... and we don't want to fail the playbook because of that, given that we'll attempt the fallback repo defined in vars.yml{,.template}.

Quoted 32 lines
Quoted 32 lines
scripts/ci/setup/gitlab-runner.yml new file +32 -0
+      register: gitlab_runner_available_archive
+
+    - name: Update base url
+      set_fact:
+        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
+      when: gitlab_runner_available_archive.status == 200
+    - debug:
+        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
+
+    - name: Set OS name (FreeBSD)
+      set_fact:
+        gitlab_runner_os: freebsd
+      when: "ansible_facts['system'] == 'FreeBSD'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
Quoted 8 lines

scripts/ci/setup/gitlab-runner.yml line 57

Quoted 8 lines
scripts/ci/setup/gitlab-runner.yml new file +8 -0
+    - name: Downloads the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+

Here, the actual download is performed, and no 40x is considered OK. I hope that explains it.

Cheers,
- Cleber.

#21 lore raw

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

Quoted 11 lines · Cleber Rosa
Quoted 10 lines · Erik Skultety
Quoted 8 lines · Cleber Rosa
scripts/ci/setup/gitlab-runner.yml new file +8 -0
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403
Quoted 8 lines
Quoted 2 lines

Why is 403 an acceptable success status code?

I missed this question in my last reply, sorry.

s3 will throw a 403 if the URI doesn't exist... and we don't want to fail the playbook because of that, given that we'll attempt the fallback repo defined in vars.yml{,.template}.

Right, forgot about the repo in vars.yml fallback, thanks for explaining.

Regards,
Erik

#22 lore raw

PATCH v4 4/4 Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa

The QEMU project has two machines (aarch64 and s390) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
 1 file changed, 192 insertions(+)
.gitlab-ci.d/custom-runners.yml +192 -0
index 3004da2bda..5b51d1b336 100644
@@ -12,3 +12,195 @@
 # strategy.
 variables:
   GIT_SUBMODULE_STRATEGY: recursive
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
Signature · 1 line
2.25.4
Reviewed-by on v4 4/4
#23 lore raw

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 8 lines · Cleber Rosa

The QEMU project has two machines (aarch64 and s390) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
 .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
 1 file changed, 192 insertions(+)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#24 lore raw

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

In reply to Cleber Rosa
Quoted 1 line · Cleber Rosa

The QEMU project has two machines (aarch64 and s390) that can be used

"s390x" =)

Quoted 7 lines

for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose.

---
  .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
  1 file changed, 192 insertions(+)
#25 lore raw

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

Quoted 5 lines · Philippe Mathieu-Daudé
Quoted 1 line · Cleber Rosa

The QEMU project has two machines (aarch64 and s390) that can be used

"s390x" =)

Sure!

Thanks!
- Cleber.

#26 lore raw
In reply to Cleber Rosa
Quoted 6 lines · Cleber Rosa

TL;DR: this should allow the QEMU maintainer to push to the staging branch, and have custom jobs running on the project's aarch64 and s390x machines. Simple usage looks like:

git push remote staging ./scripts/ci/gitlab-pipeline-status --verbose --wait

Hi Cleber,

since the QEMU project is now using the gitlab repo as main repository, it should now be possible to finally enable these custom runners, too, I think.

Could you elaborate on what's left to do to get the s390x and aarch64 runners enabled, and rebase your patches if necessary?

Thanks,
Thomas

#27 lore raw
Quoted 5 lines · Thomas Huth
Quoted 4 lines · Cleber Rosa

TL;DR: this should allow the QEMU maintainer to push to the staging branch, and have custom jobs running on the project's aarch64 and s390x machines. Simple usage looks like:

Quoted 8 lines
Quoted 2 lines

git push remote staging ./scripts/ci/gitlab-pipeline-status --verbose --wait

Hi Cleber,

since the QEMU project is now using the gitlab repo as main repository, it should now be possible to finally enable these custom runners, too, I think.

I couldn't agree more!

Quoted 3 lines

Could you elaborate on what's left to do to get the s390x and aarch64 runners enabled, and rebase your patches if necessary?

Basically what's needed is to enable the jobs, meaning committing the relevant snippets to the "gitlab-ci.{yml,d/*}" files.

Both ansible playbooks have been run on both machines, so the gitlab-runner and tooling to build QEMU are in place. The gitlab-runners have also been registered on the "qemu-project/qemu" GitLab repo.

I'll send a rebased/updated version of the playbooks, gitlab-ci.yml configurations, etc. In it, I'll propose that we enable the jobs, but ignore errors for now (allow_failure: true), because it'll ease the review. Once the jobs prove mature, waiving of failures can be removed.

How does that sound?

Thanks,
- Cleber.

Quoted 2 lines

Thanks,
Thomas

#28 lore raw
Quoted 1 line · Cleber Rosa

On Thu, Jan 28, 2021 at 12:51:20PM +0100, Thomas Huth wrote:

[...]

Quoted 10 lines
Quoted 2 lines

Could you elaborate on what's left to do to get the s390x and aarch64 runners enabled, and rebase your patches if necessary

Basically what's needed is to enable the jobs, meaning committing the relevant snippets to the "gitlab-ci.{yml,d/*}" files.

Both ansible playbooks have been run on both machines, so the gitlab-runner and tooling to build QEMU are in place. The gitlab-runners have also been registered on the "qemu-project/qemu" GitLab repo.

Quoted 7 lines

I'll send a rebased/updated version of the playbooks, gitlab-ci.yml configurations, etc. In it, I'll propose that we enable the jobs, but ignore errors for now (allow_failure: true), because it'll ease the review. Once the jobs prove mature, waiving of failures can be removed.

How does that sound?

Sounds good, please go ahead!

Thanks,
Thomas