Skip to contentSkip to conversation

QEMU Gating CI

PATCH Superseded Cleber Rosa 9 participants 57 messages Mar 12, 2020 – Jul 2, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover QEMU Gating CI

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
tests/docker: add CentOS 8 Dockerfile
tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts

 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
 docs/devel/testing.rst                        | 142 ++++++++++++
 tests/docker/dockerfiles/centos8.docker       |  32 +++
 .../dockerfiles/debian-win32-cross.docker     |   2 +-
 10 files changed, 751 insertions(+), 4 deletions(-)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
 create mode 100644 tests/docker/dockerfiles/centos8.docker
Signature · 1 line
2.24.1
#1 lore raw

PATCH 1/5 tests/docker: add CentOS 8 Dockerfile

Which is currenly missing, and will be referenced later in the contributed CI playbooks.

---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker
tests/docker/dockerfiles/centos8.docker new file +32 -0
new file mode 100644
index 0000000000..bfa0d33c9c
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt
Signature · 1 line
2.24.1
#2 lore raw

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

Quoted 37 lines · Cleber Rosa

Which is currenly missing, and will be referenced later in the contributed CI playbooks.

---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker
tests/docker/dockerfiles/centos8.docker new file +22 -0
new file mode 100644
index 0000000000..bfa0d33c9c
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \

tests/docker/dockerfiles/centos8.docker line 32

Quoted 10 lines
tests/docker/dockerfiles/centos8.docker new file +10 -0
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt

How is the packages.txt consumed later?

Erik

#3 lore raw

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

Quoted 42 lines · Erik Skultety
Quoted 41 lines · Cleber Rosa

Which is currenly missing, and will be referenced later in the contributed CI playbooks.

---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker
tests/docker/dockerfiles/centos8.docker new file +26 -0
new file mode 100644
index 0000000000..bfa0d33c9c
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
Quoted 8 lines

tests/docker/dockerfiles/centos8.docker line 32

Quoted 6 lines
tests/docker/dockerfiles/centos8.docker new file +6 -0
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt

How is the packages.txt consumed later?

I think it's just following the pattern of a number of the containers by dumping the package list for later reference by the run support scripts (tests/docker/run):

if test -n "$SHOW_ENV"; then
if test -f /packages.txt; then
echo "Packages installed:"
cat /packages.txt
echo
fi
echo "Environment variables:"
env
echo
fi

Quoted 2 lines

Erik

Signature · 1 line
Alex Bennée
#4 lore raw

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

In reply to Erik Skultety
Quoted 51 lines · Erik Skultety
Quoted 37 lines · Cleber Rosa

Which is currenly missing, and will be referenced later in the contributed CI playbooks.

---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker
tests/docker/dockerfiles/centos8.docker new file +22 -0
new file mode 100644
index 0000000000..bfa0d33c9c
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \

tests/docker/dockerfiles/centos8.docker line 32

Quoted 10 lines
tests/docker/dockerfiles/centos8.docker new file +10 -0
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt

How is the packages.txt consumed later?

Quoted 1 line

Erik

The explanation given by Alex on this thread is better than I would be able to provide. :)

Thanks,
- Cleber.

Reviewed-by on v1 1/5
#5 lore raw

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

In reply to Cleber Rosa
Quoted 4 lines · Cleber Rosa

Which is currenly missing, and will be referenced later in the contributed CI playbooks.

Quoted 43 lines
---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker
tests/docker/dockerfiles/centos8.docker new file +32 -0
new file mode 100644
index 0000000000..bfa0d33c9c
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt
Signature · 1 line
Alex Bennée
#6 lore raw

PATCH 2/5 tests/docker: make "buildah bud" output similar to "docker build"

In reply to Cleber Rosa

Podman users will most often be using buildah to build containers. Among the differences between "buildah bud|build-using-dockerfile" and a traditional "docker build" is that buildah does not run a container during build.

To the best of my knowledge and experiments, this means that runtime variables, such as ENV from one base image will not propagate into another. The end result is that the location for the cross compiler binaries, defined in the base "qemu/debian9-mxe" image, are not passed through this image. Consequently, the cross compilers are not on PATH and the build fails.

---
 tests/docker/dockerfiles/debian-win32-cross.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/docker/dockerfiles/debian-win32-cross.docker +1 -1
index 9d7053e59d..d16d6431bc 100644
@@ -9,7 +9,7 @@ MAINTAINER Philippe Mathieu-Daudé <[email protected]>
 
 ENV TARGET i686
 
-ENV PATH $PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
+ENV PATH $PATH:/usr/lib/mxe/usr/bin:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
 
 ENV PKG_CONFIG_PATH \
     $PKG_CONFIG_PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/lib/pkgconfig
Signature · 1 line
2.24.1
Acked-by on v1 2/5
#7 lore raw

Re: [PATCH 2/5] tests/docker: make "buildah bud" output similar to "docker build"

Quoted 5 lines · Cleber Rosa

Podman users will most often be using buildah to build containers. Among the differences between "buildah bud|build-using-dockerfile" and a traditional "docker build" is that buildah does not run a container during build.

Quoted 8 lines

To the best of my knowledge and experiments, this means that runtime variables, such as ENV from one base image will not propagate into another. The end result is that the location for the cross compiler binaries, defined in the base "qemu/debian9-mxe" image, are not passed through this image. Consequently, the cross compilers are not on PATH and the build fails.

Quoted 17 lines
---
 tests/docker/dockerfiles/debian-win32-cross.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/docker/dockerfiles/debian-win32-cross.docker +1 -1
index 9d7053e59d..d16d6431bc 100644
@@ -9,7 +9,7 @@ MAINTAINER Philippe Mathieu-Daudé <[email protected]>
 
 ENV TARGET i686
 
-ENV PATH $PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
+ENV PATH $PATH:/usr/lib/mxe/usr/bin:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
 
 ENV PKG_CONFIG_PATH \
     $PKG_CONFIG_PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/lib/pkgconfig
Signature · 1 line
Alex Bennée
#8 lore raw

PATCH 3/5 GitLab CI: avoid calling before_scripts on unintended jobs

In reply to Cleber Rosa

At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions.

The jobs that will be added will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition.

---
 .gitlab-ci.yml | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
.gitlab-ci.yml +14 -3
index 72f8b8aa51..3ef1551273 100644
@@ -1,11 +1,10 @@
 include:
   - local: '/.gitlab-ci-edk2.yml'
 
-before_script:
+build-system1:
+ before_script: &before_scr_apt
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
-
-build-system1:
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
       libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
@@ -18,6 +17,8 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
       libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
@@ -31,6 +32,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ before_script:
+  *before_scr_apt
  script:
  - mkdir build
  - cd build
@@ -45,6 +48,8 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq clang libgtk-3-dev libusb-dev
  - mkdir build
@@ -63,6 +68,8 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ before_script:
+  *before_scr_apt
  script:
  - mkdir build
  - cd build
@@ -72,6 +79,8 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
       xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
@@ -84,6 +93,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ before_script:
+  *before_scr_apt
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
Signature · 1 line
2.24.1
#9 lore raw

PATCH 4/5 GitLab Gating CI: introduce pipeline-status contrib script

In reply to Cleber Rosa

This script is intended to be used right after a push to a branch.

By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the --wait option to wait until the pipeline completes.

If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step.

---
 contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
contrib/ci/scripts/gitlab-pipeline-status new file +148 -0
new file mode 100755
index 0000000000..83d412daec
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+
+"""
+Checks the GitLab pipeline status for a given commit commit
+"""
+
+# pylint: disable=C0103
+
+import argparse
+import http.client
+import json
+import os
+import subprocess
+import time
+import sys
+
+
+def get_local_staging_branch_commit():
+    """
+    Returns the commit sha1 for the *local* branch named "staging"
+    """
+    result = subprocess.run(['git', 'rev-parse', 'staging'],
+                            stdin=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.DEVNULL,
+                            cwd=os.path.dirname(__file__),
+                            universal_newlines=True).stdout.strip()
+    if result == 'staging':
+        raise ValueError("There's no local staging branch")
+    if len(result) != 40:
+        raise ValueError("Branch staging HEAD doesn't look like a sha1")
+    return result
+
+
+def get_pipeline_status(project_id, commit_sha1):
+    """
+    Returns the JSON content of the pipeline status API response
+    """
+    url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id,
+                                                        commit_sha1)
+    connection = http.client.HTTPSConnection('gitlab.com')
+    connection.request('GET', url=url)
+    response = connection.getresponse()
+    if response.code != http.HTTPStatus.OK:
+        raise ValueError("Failed to receive a successful response")
+    json_response = json.loads(response.read())
+    # afaict, there should one one pipeline for the same project + commit
+    # if this assumption is false, we can add further filters to the
+    # url, such as username, and order_by.
+    if not json_response:
+        raise ValueError("No pipeline found")
+    return json_response[0]
+
+
+def wait_on_pipeline_success(timeout, interval,
+                             project_id, commit_sha):
+    """
+    Waits for the pipeline to end up to the timeout given
+    """
+    start = time.time()
+    while True:
+        if time.time() >= (start + timeout):
+            print("Waiting on the pipeline success timed out")
+            return False
+
+        status = get_pipeline_status(project_id, commit_sha)
+        if status['status'] == 'running':
+            time.sleep(interval)
+            print('running...')
+            continue
+
+        if status['status'] == 'success':
+            return True
+
+        msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url']
+        print(msg)
+        return False
+
+
+def main():
+    """
+    Script entry point
+    """
+    parser = argparse.ArgumentParser(
+        prog='pipeline-status',
+        description='check or wait on a pipeline status')
+
+    parser.add_argument('-t', '--timeout', type=int, default=7200,
+                        help=('Amount of time (in seconds) to wait for the '
+                              'pipeline to complete.  Defaults to '
+                              '%(default)s'))
+    parser.add_argument('-i', '--interval', type=int, default=60,
+                        help=('Amount of time (in seconds) to wait between '
+                              'checks of the pipeline status.  Defaults '
+                              'to %(default)s'))
+    parser.add_argument('-w', '--wait', action='store_true', default=False,
+                        help=('Wether to wait, instead of checking only once '
+                              'the status of a pipeline'))
+    parser.add_argument('-p', '--project-id', type=int, default=11167699,
+                        help=('The GitLab project ID. Defaults to the project '
+                              'for https://gitlab.com/qemu-project/qemu, that '
+                              'is, "%(default)s"'))
+    try:
+        default_commit = get_local_staging_branch_commit()
+        commit_required = False
+    except ValueError:
+        default_commit = ''
+        commit_required = True
+    parser.add_argument('-c', '--commit', required=commit_required,
+                        default=default_commit,
+                        help=('Look for a pipeline associated with the given '
+                              'commit.  If one is not explicitly given, the '
+                              'commit associated with the local branch named '
+                              '"staging" is used.  Default: %(default)s'))
+    parser.add_argument('--verbose', action='store_true', default=False,
+                        help=('A minimal verbosity level that prints the '
+                              'overall result of the check/wait'))
+
+    args = parser.parse_args()
+
+    try:
+        if args.wait:
+            success = wait_on_pipeline_success(
+                args.timeout,
+                args.interval,
+                args.project_id,
+                args.commit)
+        else:
+            status = get_pipeline_status(args.project_id,
+                                         args.commit)
+            success = status['status'] == 'success'
+    except Exception as error:      # pylint: disable=W0703
+        success = False
+        if args.verbose:
+            print("ERROR: %s" % error.args[0])
+
+    if success:
+        if args.verbose:
+            print('success')
+        sys.exit(0)
+    else:
+        if args.verbose:
+            print('failure')
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.24.1
#10 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

Quoted 23 lines · Cleber Rosa

This script is intended to be used right after a push to a branch.

By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the --wait option to wait until the pipeline completes.

If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step.

---
 contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
contrib/ci/scripts/gitlab-pipeline-status new file +0 -0
new file mode 100755
index 0000000000..83d412daec

contrib/ci/scripts/gitlab-pipeline-status line 5

Quoted 5 lines
contrib/ci/scripts/gitlab-pipeline-status new file +5 -0
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+
+"""
+Checks the GitLab pipeline status for a given commit commit
+"""

All new files, and particularly new scripts and source files, should have the usual copyright-and-license comment at the top, please.

thanks
-- PMM

#11 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

Quoted 8 lines · Peter Maydell

All new files, and particularly new scripts and source files, should have the usual copyright-and-license comment at the top, please.

thanks
-- PMM

My bad. I'll wait for further comments and included that in a v2.

Cheers,
- Cleber.

#12 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

In reply to Cleber Rosa
Quoted 9 lines · Cleber Rosa

This script is intended to be used right after a push to a branch.

By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the --wait option to wait until the pipeline completes.

If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step.

On IRC yesterday we were discussing the current .gitlab-ci.yml status, and how frustrating it is that every time we get it green, more code is soon merged that turns it red again.

It feels like it should be an easy win to declare that the current GitLab CI jobs are to be used as a gating tests for merges to master.

As & when custom runners come online, their jobs can simply augment the existing jobs. IOW, use of GitLab CI for gating master shouldn't be dependant on setup of custom runners which we've been waiting on for a long term.

Peter indicated that his main requirement is a way to automate the task of kicking off the CI job & getting its status. It seems like the script in this patch should fullfill that requirement.

Assuming Peter (or whomever is going todo the merge) has a fork of

https://gitlab.com/qemu-project/qemu

then they need to find the "ID" number of their fork. This is visible at the top for the page for their fork eg mine:

https://gitlab.com/berrange/qemu

reports:

"Project ID: 18588805 "

Assuming the fork is configured as a git remote called "gitlab", then to use GitLab CI as gating test, all that appears to be needed is

   $ git push gitlab
   $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805

If this is an acceptable level of automation for Peter, then can we get this specific patch merged right now and make current GitLab CI be gating for master.

With GitLab as gating, then we have further incentive to move all the jobs currently on Travis CI and Shippable, over to use GitLab too, and also use cirrus-run to make Cirrus CI jobs be proxied from GitLab. All this can be in parallel with adding custom GitLab runners for expanding testing coverage still further.

Quoted 164 lines
---
 contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
contrib/ci/scripts/gitlab-pipeline-status new file +148 -0
new file mode 100755
index 0000000000..83d412daec
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+
+"""
+Checks the GitLab pipeline status for a given commit commit
+"""
+
+# pylint: disable=C0103
+
+import argparse
+import http.client
+import json
+import os
+import subprocess
+import time
+import sys
+
+
+def get_local_staging_branch_commit():
+    """
+    Returns the commit sha1 for the *local* branch named "staging"
+    """
+    result = subprocess.run(['git', 'rev-parse', 'staging'],
+                            stdin=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.DEVNULL,
+                            cwd=os.path.dirname(__file__),
+                            universal_newlines=True).stdout.strip()
+    if result == 'staging':
+        raise ValueError("There's no local staging branch")
+    if len(result) != 40:
+        raise ValueError("Branch staging HEAD doesn't look like a sha1")
+    return result
+
+
+def get_pipeline_status(project_id, commit_sha1):
+    """
+    Returns the JSON content of the pipeline status API response
+    """
+    url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id,
+                                                        commit_sha1)
+    connection = http.client.HTTPSConnection('gitlab.com')
+    connection.request('GET', url=url)
+    response = connection.getresponse()
+    if response.code != http.HTTPStatus.OK:
+        raise ValueError("Failed to receive a successful response")
+    json_response = json.loads(response.read())
+    # afaict, there should one one pipeline for the same project + commit
+    # if this assumption is false, we can add further filters to the
+    # url, such as username, and order_by.
+    if not json_response:
+        raise ValueError("No pipeline found")
+    return json_response[0]
+
+
+def wait_on_pipeline_success(timeout, interval,
+                             project_id, commit_sha):
+    """
+    Waits for the pipeline to end up to the timeout given
+    """
+    start = time.time()
+    while True:
+        if time.time() >= (start + timeout):
+            print("Waiting on the pipeline success timed out")
+            return False
+
+        status = get_pipeline_status(project_id, commit_sha)
+        if status['status'] == 'running':
+            time.sleep(interval)
+            print('running...')
+            continue
+
+        if status['status'] == 'success':
+            return True
+
+        msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url']
+        print(msg)
+        return False
+
+
+def main():
+    """
+    Script entry point
+    """
+    parser = argparse.ArgumentParser(
+        prog='pipeline-status',
+        description='check or wait on a pipeline status')
+
+    parser.add_argument('-t', '--timeout', type=int, default=7200,
+                        help=('Amount of time (in seconds) to wait for the '
+                              'pipeline to complete.  Defaults to '
+                              '%(default)s'))
+    parser.add_argument('-i', '--interval', type=int, default=60,
+                        help=('Amount of time (in seconds) to wait between '
+                              'checks of the pipeline status.  Defaults '
+                              'to %(default)s'))
+    parser.add_argument('-w', '--wait', action='store_true', default=False,
+                        help=('Wether to wait, instead of checking only once '
+                              'the status of a pipeline'))
+    parser.add_argument('-p', '--project-id', type=int, default=11167699,
+                        help=('The GitLab project ID. Defaults to the project '
+                              'for https://gitlab.com/qemu-project/qemu, that '
+                              'is, "%(default)s"'))
+    try:
+        default_commit = get_local_staging_branch_commit()
+        commit_required = False
+    except ValueError:
+        default_commit = ''
+        commit_required = True
+    parser.add_argument('-c', '--commit', required=commit_required,
+                        default=default_commit,
+                        help=('Look for a pipeline associated with the given '
+                              'commit.  If one is not explicitly given, the '
+                              'commit associated with the local branch named '
+                              '"staging" is used.  Default: %(default)s'))
+    parser.add_argument('--verbose', action='store_true', default=False,
+                        help=('A minimal verbosity level that prints the '
+                              'overall result of the check/wait'))
+
+    args = parser.parse_args()
+
+    try:
+        if args.wait:
+            success = wait_on_pipeline_success(
+                args.timeout,
+                args.interval,
+                args.project_id,
+                args.commit)
+        else:
+            status = get_pipeline_status(args.project_id,
+                                         args.commit)
+            success = status['status'] == 'success'
+    except Exception as error:      # pylint: disable=W0703
+        success = False
+        if args.verbose:
+            print("ERROR: %s" % error.args[0])
+
+    if success:
+        if args.verbose:
+            print('success')
+        sys.exit(0)
+    else:
+        if args.verbose:
+            print('failure')
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.24.1

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#13 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

Quoted 8 lines · Daniel P. Berrangé
Quoted 7 lines · Cleber Rosa

This script is intended to be used right after a push to a branch.

By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the --wait option to wait until the pipeline completes.

Quoted 7 lines
Quoted 2 lines

If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step.

On IRC yesterday we were discussing the current .gitlab-ci.yml status, and how frustrating it is that every time we get it green, more code is soon merged that turns it red again.

Hi Daniel,

I know this is nothing new to you, but "green" turning "red" is something that can be minimized, but never completely abolished. We've had discussions on how we could minimize those, and suggestions on how to address those occurrences. The points that I remember about it are:

1. For jobs supposed to run on containers and vms, use images prepared to run the builds and tests, that is, don't do the prep steps (package installations and others) within the job itself - this is currently missing.

2. For bare metal, have the setup either documented or scripted and run before the jobs (in the patch series I sent, these were done in a number of Ansible based playbooks).

3. Have a mechanism for promoting non-gating jobs to gating jobs. Keeping track of their reliability over a period of time is a way to forecast it future behavior, as much as it can be done.

4. Have a MAINTAINERS entry for those jobs, and have maintainers attemtp to address issues within a specified amount of time; if that fails, have a mechanism to downgrade those jobs to non-gating. A patch removing/moving entries from .gitlab-ci*.yml would do that and of course sping another set of CI jobs bringing things back to green.

Quoted 3 lines

It feels like it should be an easy win to declare that the current GitLab CI jobs are to be used as a gating tests for merges to master.

I'm fine with that approach if most people agree. This would mean point #1 is ignored for now. For everyone's sake, points #3 and #4 are a must IMO. Point #2 will applied as we introduced our non-shared gitlab runners.

Quoted 33 lines

As & when custom runners come online, their jobs can simply augment the existing jobs. IOW, use of GitLab CI for gating master shouldn't be dependant on setup of custom runners which we've been waiting on for a long term.

Peter indicated that his main requirement is a way to automate the task of kicking off the CI job & getting its status. It seems like the script in this patch should fullfill that requirement.

Assuming Peter (or whomever is going todo the merge) has a fork of

https://gitlab.com/qemu-project/qemu

then they need to find the "ID" number of their fork. This is visible at the top for the page for their fork eg mine:

https://gitlab.com/berrange/qemu

reports:

   "Project ID: 18588805 "

Assuming the fork is configured as a git remote called "gitlab", then to use GitLab CI as gating test, all that appears to be needed is

   $ git push gitlab
   $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805

If this is an acceptable level of automation for Peter, then can we get this specific patch merged right now and make current GitLab CI be gating for master.

Quoted 6 lines

With GitLab as gating, then we have further incentive to move all the jobs currently on Travis CI and Shippable, over to use GitLab too, and also use cirrus-run to make Cirrus CI jobs be proxied from GitLab. All this can be in parallel with adding custom GitLab runners for expanding testing coverage still further.

The script should indeed work with the workflow you described. I'll just run a few tests, to make sure that are no caveats, and I'll let you and Peter know about it.

Thanks,
- Cleber.

#14 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

Quoted 5 lines · Cleber Rosa

The script should indeed work with the workflow you described. I'll just run a few tests, to make sure that are no caveats, and I'll let you and Peter know about it.

FIY, I've cherry-pick just the patch with the scripts (for no reason other than having some content) and pushed it to a different branch than staging:

https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners

Then I ran:

  ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8

Which worked as expected. BTW, the commit wouldn't be necessary if I was checking on a "staging" branch. The pipeline triggered (and being waited on by the script is at):

https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485

So I believe it's now a matter of sorting out job the inclusion/exclusion aspects, that is, if we start with all/some of the current jobs and some or none of the jobs intended to run on the s390/aarch64/x86_64 (non-shared) runners.

Thanks,
- Cleber.

#15 lore raw

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

Quoted 10 lines · Cleber Rosa
Quoted 5 lines · Cleber Rosa

The script should indeed work with the workflow you described. I'll just run a few tests, to make sure that are no caveats, and I'll let you and Peter know about it.

FIY, I've cherry-pick just the patch with the scripts (for no reason other than having some content) and pushed it to a different branch than staging:

Quoted 7 lines

https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners

Then I ran:

  ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8

Which worked as expected.

Great news!

Can you respin as a new series with this as single patch, and explanation in the cover about how to setup it? This is the last piece missing for Peter to use the effort done by Alex/Daniel/Thomas on the GitLab side :)

Quoted 14 lines

BTW, the commit wouldn't be necessary if I
was checking on a "staging" branch. The pipeline triggered (and being waited on by the script is at):

   https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485

So I believe it's now a matter of sorting out job the inclusion/exclusion aspects, that is, if we start with all/some of the current jobs and some or none of the jobs intended to run on the s390/aarch64/x86_64 (non-shared) runners.

Thanks,
- Cleber.

#16 lore raw

PATCH 5/5 GitLab Gating CI: initial set of jobs, documentation and scripts

In reply to Cleber Rosa

This is a mapping of Peter's "remake-merge-builds" and "pull-buildtest" scripts, gone through some updates, adding some build option and removing others.

To add to some of the diversity of platforms that QEMU covers, this initial set of jobs require three machines:

- Ubuntu 18.04
- Fedora 30
- FreeBSD 12.1

All jobs have been marked as x86_64, but others can be introduced with different architectures. During the development of this set of jobs, the GitLab CI was tested with many other architectures, including ppc64, s390x and aarch64.

More information can be found in the documentation itself.

---
 .gitlab-ci-gating.yml                      | 111 +++++++++++
 .gitlab-ci.yml                             |  15 ++
 contrib/ci/orgs/qemu/build-environment.yml | 208 +++++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml     |  65 +++++++
 contrib/ci/orgs/qemu/inventory             |   2 +
 contrib/ci/orgs/qemu/vars.yml              |  13 ++
 docs/devel/testing.rst                     | 142 ++++++++++++++
 7 files changed, 556 insertions(+)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml
.gitlab-ci-gating.yml new file +111 -0
new file mode 100644
index 0000000000..4c6010bcc4
@@ -0,0 +1,111 @@
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
+
+# All ubuntu-18.04.3 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04"
+ubuntu-18.04.3-x86_64-all-linux-static:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - ./configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04.3-x86_64-all:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-alldbg:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-clang:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-tci:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04.3-x86_64-notcg:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All fedora-30 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Fedora 30"
+fedora-30-x86_64-cross-w32:
+ tags:
+ - fedora_30
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --cross-prefix=i686-w64-mingw32-
+ - make --output-sync -j`nproc`
+
+fedora-30-x86_64-cross-w64:
+ tags:
+ - fedora_30
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --cross-prefix=x86_64-w64-mingw32-
+ - make --output-sync -j`nproc`
+
+# All freebsd_12.1 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on FreeBSD 12.x"
+freebsd-12.1-x86_64-all:
+ tags:
+ - freebsd_12.1
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure
+ - gmake --output-sync -j`sysctl -n hw.ncpu`
+ - gmake --output-sync -j`sysctl -n hw.ncpu` check V=1
.gitlab-ci.yml +15 -0
index 3ef1551273..8c51672dc1 100644
@@ -1,7 +1,10 @@
 include:
   - local: '/.gitlab-ci-edk2.yml'
+  - local: '/.gitlab-ci-gating.yml'
 
 build-system1:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script: &before_scr_apt
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
@@ -17,6 +20,8 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -32,6 +37,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -48,6 +55,8 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -68,6 +77,8 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -79,6 +90,8 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -93,6 +106,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
contrib/ci/orgs/qemu/build-environment.yml new file +208 -0

208 lines not on this page · raw message

contrib/ci/orgs/qemu/gitlab-runner.yml new file +65 -0

65 lines not on this page · raw message

contrib/ci/orgs/qemu/inventory new file +2 -0
new file mode 100644
index 0000000000..8bb7ba6b33
@@ -0,0 +1,2 @@
+[local]
+localhost
contrib/ci/orgs/qemu/vars.yml new file +13 -0
new file mode 100644
index 0000000000..0095851172
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 12.8.0
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
docs/devel/testing.rst +142 -0

145 lines not on this page · raw message

Signature · 1 line
2.24.1
#17 lore raw
In reply to Cleber Rosa
Quoted 4 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

OK, so my question here is:
* what are the instructions that I have to follow to be
able to say "ok, here's my branch, run it through these tests, please" ?

thanks
-- PMM

#18 lore raw
Quoted 10 lines · Peter Maydell
Quoted 4 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

OK, so my question here is:
* what are the instructions that I have to follow to be
able to say "ok, here's my branch, run it through these tests, please" ?

The quick answer is:

 $ git push [email protected]:qemu-project/qemu.git my-branch:staging

The longer explanation is that these jobs are limited to a "staging" branch, so all you'd have to do is to push something to a branch called "staging". If that branch happens to be from the "gitlab.com/qemu-project/qemu" repo, than the runners setup there would be used. The documentation an ansible playbooks are supposed to help with this setup.

Once that push happens, you could use:

 $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait

Before doing something like:

 $ git push [email protected]:qemu-project/qemu.git my-branch:master
Quoted 4 lines

thanks
-- PMM

Let me know if that makes sense.

Cheers,
- Cleber.

#19 lore raw
Quoted 9 lines · Cleber Rosa
Quoted 4 lines · Peter Maydell

OK, so my question here is:
* what are the instructions that I have to follow to be
able to say "ok, here's my branch, run it through these tests, please" ?

The quick answer is:

Quoted 8 lines
 $ git push [email protected]:qemu-project/qemu.git my-branch:staging

The longer explanation is that these jobs are limited to a "staging" branch, so all you'd have to do is to push something to a branch called "staging". If that branch happens to be from the "gitlab.com/qemu-project/qemu" repo, than the runners setup there would be used. The documentation an ansible playbooks are supposed to help with this setup.

Great, thanks. Could I do that for testing purposes with a staging branch that includes these patches, or would we have to wait for them to be in master before it works?

-- PMM

#20 lore raw
Quoted 19 lines · Peter Maydell
Quoted 9 lines · Cleber Rosa
Quoted 4 lines · Peter Maydell

OK, so my question here is:
* what are the instructions that I have to follow to be
able to say "ok, here's my branch, run it through these tests, please" ?

The quick answer is:

Quoted 8 lines
 $ git push [email protected]:qemu-project/qemu.git my-branch:staging

The longer explanation is that these jobs are limited to a "staging" branch, so all you'd have to do is to push something to a branch called "staging". If that branch happens to be from the "gitlab.com/qemu-project/qemu" repo, than the runners setup there would be used. The documentation an ansible playbooks are supposed to help with this setup.

Quoted 4 lines

Great, thanks. Could I do that for testing purposes with a staging branch that includes these patches, or would we have to wait for them to be in master before it works?

You can definitely do that with a staging branch that includes these patches.

And with regards to setting up machines with runners and other requirements, I tried to make it easy to replicate, but YMMV. So, please let me know if you find any issues whatsoever.

Quoted 2 lines

-- PMM

Thanks,
- Cleber.

#21 lore raw
In reply to Cleber Rosa
Quoted 3 lines · Cleber Rosa

The quick answer is:

 $ git push [email protected]:qemu-project/qemu.git my-branch:staging

So I did this bit...

Quoted 3 lines

Once that push happens, you could use:

 $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait

...but this script just says:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait
ERROR: No pipeline found
failure

thanks
-- PMM

#22 lore raw

----- Original Message -----

Quoted 18 lines

From: "Peter Maydell" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Alex Bennée" <[email protected]>, "QEMU Developers" <[email protected]>, "Fam Zheng" <[email protected]>, "Eduardo Habkost" <[email protected]>, "Beraldo Leal" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Thomas Huth" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Erik Skultety" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Wainer Moschetta" <[email protected]>
Sent: Monday, March 16, 2020 7:57:33 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 3 lines · Cleber Rosa

The quick answer is:

 $ git push [email protected]:qemu-project/qemu.git my-branch:staging

So I did this bit...

Quoted 2 lines

Once that push happens, you could use:

Quoted 8 lines
Quoted 1 line
 $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait

...but this script just says:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait
ERROR: No pipeline found
failure

Hi Peter,

A few possible reasons come to my mind:

1) It usually takes a few seconds after the push for the pipeline to

2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, you'd have to tweak the project ID (-p|--project-id).

3) The local branch is not called "staging", so the script can not find the commit ID, in that case you can use -c|--commit.

Quoted 4 lines

thanks
-- PMM

Please let me know if any of these points helps.

Cheers,
- Cleber.

#23 lore raw
Quoted 9 lines · Cleber Rosa

A few possible reasons come to my mind:

1) It usually takes a few seconds after the push for the pipeline to

2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, you'd have to tweak the project ID (-p|--project-id).

3) The local branch is not called "staging", so the script can not find the commit ID, in that case you can use -c|--commit.

Yes, the local branch is something else for the purposes of testing this series. But using --commit doesn't work either:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
81beaaab0851fe8c4db971 --wait
ERROR: No pipeline found
failure

On the web UI:
https://gitlab.com/qemu-project/qemu/pipelines
the pipelines are marked "stuck" (I don't know why there are two of them for the same commit); drilling down, the build part has completed but all the test parts are pending with "This job is stuck because you don't have any active runners online with any of these tags assigned to them" type messages.

thanks
-- PMM

#24 lore raw

----- Original Message -----

Quoted 20 lines

From: "Peter Maydell" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Alex Bennée" <[email protected]>, "QEMU Developers" <[email protected]>, "Fam Zheng" <[email protected]>, "Eduardo Habkost" <[email protected]>, "Beraldo Leal" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Thomas Huth" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Erik Skultety" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Wainer Moschetta" <[email protected]>
Sent: Monday, March 16, 2020 8:12:16 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 9 lines · Cleber Rosa

A few possible reasons come to my mind:

1) It usually takes a few seconds after the push for the pipeline to

2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, you'd have to tweak the project ID (-p|--project-id).

3) The local branch is not called "staging", so the script can not find the commit ID, in that case you can use -c|--commit.

Quoted 8 lines

Yes, the local branch is something else for the purposes of testing this series. But using --commit doesn't work either:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
81beaaab0851fe8c4db971 --wait
ERROR: No pipeline found
failure

This may be a bug in the script because I was not expecting two pipelines for the same commit. Checking...

Quoted 9 lines

On the web UI:
https://gitlab.com/qemu-project/qemu/pipelines
the pipelines are marked "stuck" (I don't know why there are two of them for the same commit); drilling down, the build part has completed but all the test parts are pending with "This job is stuck because you don't have any active runners online with any of these tags assigned to them" type messages.


I had also not come across the duplicate pipelines, so I'm trying to understand what's that about.

About the runners and the fact that the job is stuck without them, the message seems straightforward enough, but I can't get to the project configuration to look at the registered runners with my current permissions (set as "developer").

Quoted 4 lines

thanks
-- PMM

Thanks,
- Cleber.

#25 lore raw

----- Original Message -----

Quoted 41 lines

From: "Cleber Rosa" <[email protected]>
To: "Peter Maydell" <[email protected]>
Cc: "Alex Bennée" <[email protected]>, "QEMU Developers" <[email protected]>, "Fam Zheng" <[email protected]>, "Eduardo Habkost" <[email protected]>, "Beraldo Leal" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Thomas Huth" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Erik Skultety" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Wainer Moschetta" <[email protected]>
Sent: Monday, March 16, 2020 8:26:46 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

----- Original Message -----

Quoted 29 lines

From: "Peter Maydell" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Alex Bennée" <[email protected]>, "QEMU Developers" <[email protected]>, "Fam Zheng" <[email protected]>,
"Eduardo Habkost" <[email protected]>, "Beraldo Leal" <[email protected]>, "Philippe Mathieu-Daudé"
<[email protected]>, "Thomas Huth" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Erik
Skultety" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Wainer Moschetta" <[email protected]>
Sent: Monday, March 16, 2020 8:12:16 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 11 lines · Cleber Rosa

A few possible reasons come to my mind:

1) It usually takes a few seconds after the push for the pipeline to

2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu,
you'd have to tweak the project ID (-p|--project-id).

3) The local branch is not called "staging", so the script can not find the
commit ID, in that case you can use -c|--commit.

Yes, the local branch is something else for the purposes of testing this series. But using --commit doesn't work either:

Quoted 9 lines
Quoted 5 lines
$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
81beaaab0851fe8c4db971 --wait
ERROR: No pipeline found
failure

This may be a bug in the script because I was not expecting two pipelines for the same commit. Checking...

Looks like the GitLab API requires a 40 char commit ID, so this should work:

 $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit 81beaaab0851fe8c4db971df555600152bb83a6c --wait

It'll stay silent at first, but then will print a message every 60 seconds.

Thanks,
- Cleber.

#26 lore raw
In reply to Cleber Rosa
Quoted 4 lines · Cleber Rosa

About the runners and the fact that the job is stuck without them, the message seems straightforward enough, but I can't get to the project configuration to look at the registered runners with my current permissions (set as "developer").

I've moved you up to 'maintainer' status, hopefully that is sufficient to look at the relevant config ?

thanks
-- PMM

#27 lore raw

----- Original Message -----

Quoted 15 lines

From: "Peter Maydell" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Monday, March 16, 2020 10:57:30 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 4 lines · Cleber Rosa

About the runners and the fact that the job is stuck without them, the message seems straightforward enough, but I can't get to the project configuration to look at the registered runners with my current permissions (set as "developer").

Quoted 7 lines

I've moved you up to 'maintainer' status, hopefully that is sufficient to look at the relevant config ?

thanks
-- PMM

Hi Peter,

Yes, that did the trick and I can now see the configuration. What I can *not* see is any "Specific Runner" configured. So maybe:

1) The documentation I included is not clear enough about the fact that setup steps need to be done on a machine so that it becomes a "Runner"

2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml) is not working as intended

3) Some expectations misalignment on machines that would be available to run those jobs

In any case, none of those should be big problems. Please let me know what you did/experienced/expected up to this point, and we can continue from there.

Regards,
- Cleber.

#28 lore raw
Quoted 6 lines · Cleber Rosa

Yes, that did the trick and I can now see the configuration. What I can *not* see is any "Specific Runner" configured. So maybe:

1) The documentation I included is not clear enough about the fact that setup steps need to be done on a machine so that it becomes a "Runner"

Quoted 8 lines

2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml) is not working as intended

3) Some expectations misalignment on machines that would be available to run those jobs

In any case, none of those should be big problems. Please let me know what you did/experienced/expected up to this point, and we can continue from there.

Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup. I thought all the stuff about "how to set up runners" was only for people who wanted to set up some 3rd-party CI for non-official forks or for when we wanted to add extra runners in future (eg for architectures not yet covered). So the only thing I did was follow your "just push to staging" instructions.

thanks
-- PMM

#29 lore raw
Quoted 13 lines · Peter Maydell
Quoted 12 lines · Cleber Rosa

Yes, that did the trick and I can now see the configuration. What I can *not* see is any "Specific Runner" configured. So maybe:

1) The documentation I included is not clear enough about the fact that setup steps need to be done on a machine so that it becomes a "Runner"

2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml) is not working as intended

3) Some expectations misalignment on machines that would be available to run those jobs

Quoted 6 lines
Quoted 2 lines

In any case, none of those should be big problems. Please let me know what you did/experienced/expected up to this point, and we can continue from there.

Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup.

Hi Peter,

So, I had to use temporary hardware resources to set this up (and set it up countless times TBH). I had the understanding based on the list of machines you documented[1] that at least some of them would be used for the permanent setup.

Quoted 6 lines

I thought all the stuff about "how to set up runners" was only for people who wanted to set up some 3rd-party CI for non-official forks or for when we wanted to add extra runners in future (eg for architectures not yet covered). So the only thing I did was follow your "just push to staging" instructions.

OK, I see it, now it makes more sense. So we're "only" missing the setup for the machines we'll use for the more permanent setup. Would you like to do a staged setup/migration using one or some of the machines you documented? I'm 100% onboard to help with this, meaning that I can assist you with instructions, or do "pair setup" of the machines if needed. I think a good part of the evaluation here comes down to how manageable/reproducible the setup is, so it'd make sense for one to be part of the setup itself.

FIY there's also the possibility of grabbing some free VMs on GCP, Azure, etc and setting them up as GitLab runners in a temporary way (because of the temporarily free and VM nature). I have a few problems with this approach, including the fact that it doesn't yield the complete experience wrt using hardware one owns and will have to manage, besides the hardware limitations themselves.

Please let me know how you want to move on from here.

Cheers,
- Cleber.

Quoted 3 lines

thanks
-- PMM

[1] https://wiki.qemu.org/Requirements/GatingCI#Current_Tests

#30 lore raw
Quoted 10 lines · Cleber Rosa
Quoted 3 lines · Peter Maydell

Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup.

So, I had to use temporary hardware resources to set this up (and set it up countless times TBH). I had the understanding based on the list of machines you documented[1] that at least some of them would be used for the permanent setup.

Well, some of them will be (eg the s390 box), but some of them are my personal ones that can't be reused easily. I'd assumed in any case that gitlab would have at least support for x86 hosts:
we are definitely not going to continue to use my desktop machine for running CI builds! Also IIRC RedHat said they'd be able to provide some machines for runners.

Quoted 8 lines

OK, I see it, now it makes more sense. So we're "only" missing the setup for the machines we'll use for the more permanent setup. Would you like to do a staged setup/migration using one or some of the machines you documented? I'm 100% onboard to help with this, meaning that I can assist you with instructions, or do "pair setup" of the machines if needed. I think a good part of the evaluation here comes down to how manageable/reproducible the setup is, so it'd make sense for one to be part of the setup itself.

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

But to a large degree I really don't want to have to get into the details of how gitlab works or setting up runners myself if I can avoid it. We're going through this migration because I want to be able to hand off the CI stuff to other people, not to retain control of it.

thanks
-- PMM

#31 lore raw
Quoted 12 lines · Peter Maydell
Quoted 10 lines · Cleber Rosa
Quoted 3 lines · Peter Maydell

Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup.

So, I had to use temporary hardware resources to set this up (and set it up countless times TBH). I had the understanding based on the list of machines you documented[1] that at least some of them would be used for the permanent setup.

Quoted 6 lines

Well, some of them will be (eg the s390 box), but some of them are my personal ones that can't be reused easily. I'd assumed in any case that gitlab would have at least support for x86 hosts:
we are definitely not going to continue to use my desktop machine for running CI builds! Also IIRC RedHat said they'd be able to provide some machines for runners.

Correct! As discussed at the QEMU summit, we'll gladly chip in runners to test the stuff we care about, but to match the coverage of your private zoo of machines, others will have to chip in, too.

Quoted 9 lines
Quoted 8 lines

OK, I see it, now it makes more sense. So we're "only" missing the setup for the machines we'll use for the more permanent setup. Would you like to do a staged setup/migration using one or some of the machines you documented? I'm 100% onboard to help with this, meaning that I can assist you with instructions, or do "pair setup" of the machines if needed. I think a good part of the evaluation here comes down to how manageable/reproducible the setup is, so it'd make sense for one to be part of the setup itself.

Quoted 6 lines

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Does this sound workable?

Quoted 5 lines

But to a large degree I really don't want to have to get into the details of how gitlab works or setting up runners myself if I can avoid it. We're going through this migration because I want to be able to hand off the CI stuff to other people, not to retain control of it.

Understand. We need contributions to gating CI, but the whole point of this exercise is to make people other than *you* contribute to our gating CI :)

Let me use this opportunity to say thank you for all your integration work!

#32 lore raw
Quoted 14 lines · Markus Armbruster
Quoted 12 lines · Peter Maydell
Quoted 10 lines · Cleber Rosa
Quoted 3 lines · Peter Maydell

Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup.

So, I had to use temporary hardware resources to set this up (and set it up countless times TBH). I had the understanding based on the list of machines you documented[1] that at least some of them would be used for the permanent setup.

Quoted 6 lines
Quoted 6 lines

Well, some of them will be (eg the s390 box), but some of them are my personal ones that can't be reused easily. I'd assumed in any case that gitlab would have at least support for x86 hosts:
we are definitely not going to continue to use my desktop machine for running CI builds! Also IIRC RedHat said they'd be able to provide some machines for runners.

While GitLab let's you run x86 code for free with the "Linux Shared Runners"[1], I don't think it would be suitable to what we're trying to achieve. It's limited to a single OS (CoreOS), single architecture and really geared towards running containers. BTW, if it isn't clear, this is the approach being used today for the jobs defined on ".gitlab-ci.yml".

IMO we can leverage and still expand on the use of the "Linux Shared Runners", but to really get a grasp oh how well this model can work for QEMU, we'll need "Specific Runners", because we're validating how/if we can depend on it for OS/architectures they don't support on shared runners (and sometimes not even for the gitlab-runner agent).

Quoted 5 lines

Correct! As discussed at the QEMU summit, we'll gladly chip in runners to test the stuff we care about, but to match the coverage of your private zoo of machines, others will have to chip in, too.

I'm sorry I missed the original discussions, and I'm even more sorry if that led to any misunderstandings here.

Quoted 27 lines
Quoted 9 lines
Quoted 8 lines

OK, I see it, now it makes more sense. So we're "only" missing the setup for the machines we'll use for the more permanent setup. Would you like to do a staged setup/migration using one or some of the machines you documented? I'm 100% onboard to help with this, meaning that I can assist you with instructions, or do "pair setup" of the machines if needed. I think a good part of the evaluation here comes down to how manageable/reproducible the setup is, so it'd make sense for one to be part of the setup itself.

Quoted 6 lines

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Right, while we've tried to streamline the process of setting up the machines, there will be plenty of changes to improve the automation.

More importantly, maintaining the machines is very important to the super important goal of catching code regressions only, and not facing other failures. Mundane tasks such as making sure enough disk space is always available can be completely change the perception of the usefulness of a CI environment. And for this maintenance, we need help from people "owning" those machines.

Quoted 9 lines

Does this sound workable?

Quoted 5 lines

But to a large degree I really don't want to have to get into the details of how gitlab works or setting up runners myself if I can avoid it. We're going through this migration because I want to be able to hand off the CI stuff to other people, not to retain control of it.

Quoted 8 lines

Understand. We need contributions to gating CI, but the whole point of this exercise is to make people other than *you* contribute to our gating CI :)

Let me use this opportunity to say thank you for all your integration work!

^ THIS. I have to say that I'm still amazed as to how Peter has managed to automate, integrate and run all those tests in such varied environments for so long. Major kudos!

- Cleber.

[1] https://docs.gitlab.com/ee/user/gitlab_com/#linux-shared-runners

#33 lore raw
In reply to Markus Armbruster
Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

thanks
-- PMM

#34 lore raw

----- Original Message -----

Quoted 39 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

From now on, I'll give you an update every time a significant event occurs on our side.

Quoted 4 lines

thanks
-- PMM

Thanks for checking in!
- Cleber.

#35 lore raw
Quoted 52 lines · Cleber Rosa

----- Original Message -----

Quoted 39 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 5 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#36 lore raw

----- Original Message -----

Quoted 72 lines

From: "Daniel P. Berrangé" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Peter Maydell" <[email protected]>, "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Wainer Moschetta" <[email protected]>, "Markus Armbruster" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Alex Bennée" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Thursday, April 23, 2020 1:13:22 PM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 61 lines · Cleber Rosa

----- Original Message -----

Quoted 46 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik
Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>,
"QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo
Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 24 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these
guys.

Quoted 5 lines

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and
the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Hi Peter,

Quoted 8 lines
Quoted 5 lines

You hit the nail in the head here. We were affected indeed with our ability
to move some machines from one lab to another (across the country), but we're
actively working on it.

For x86, do we really need to be using custom runners ?

Hi Daniel,

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

My experiments were really around that point, I mean, confirming that we can grow the number of architectures/runners/jobs/configurations to provide a coverage equal or greater to what Peter already does.

Quoted 5 lines

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

They will continue to be able use the jobs and runners already defined in the .gitlab-ci.yml file. This work will only affect people pushing to the/a "staging" branch.

Quoted 4 lines

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

If you're looking for symmetry between any PR and "merge time" jobs, the only solution is to allow any PR to access all the diverse set of non-shared machines we're hoping to have in the near future. This may be something we'll get to, but I doubt we can tackle it in the near future now.

Quoted 4 lines

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Quoted 9 lines

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

We've discussed this before at the RFC time, wrt how the goal is for a wider community to provide a wider range of jobs. Even for x86, one may want to require their jobs to run on a given accelerator, such as KVM, so we need to consider that from the very beginning.

I don't see a problem with converging jobs with are being run on custom runners back into shared runners as much as possible. At the RFC discussion, I actually pointed out how the build phase could be running essentially on pre-built containers (on shared runners), but the test phase, say testing KVM, should not be bound to that.

So in essence, right now, moving everything to containers would invalidate the exercise of being able to care for those custom architectures/builds/jobs we'll need in the near future. And that's really the whole point here.

Cheers,
- Cleber.

Quoted 11 lines

Regards,
Daniel
--

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
|:|
|: https://libvirt.org         -o-            https://fstop138.berrange.com
|:|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
|:|
#37 lore raw
Quoted 5 lines · Cleber Rosa

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

We do have the S390 machine that IBM kindly made available to the project -- that is not a personal or Linaro machine, so there are no issues with giving you a login on that so you can set it up as a CI runner. Drop me an email if you want access to it.

thanks
-- PMM

#38 lore raw
Quoted 8 lines · On Thu, 23 Apr 2020 18:50:47 +0100 Peter Maydell <[email protected]>…
Quoted 6 lines · Cleber Rosa

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

Quoted 6 lines

We do have the S390 machine that IBM kindly made available to the project -- that is not a personal or Linaro machine, so there are no issues with giving you a login on that so you can set it up as a CI runner. Drop me an email if you want access to it.

I sure do. I'll shoot you an email off-list.

Quoted 3 lines

thanks
-- PMM

Thanks,
- Cleber.

#39 lore raw
In reply to Cleber Rosa
Quoted 92 lines · Cleber Rosa

----- Original Message -----

Quoted 72 lines

From: "Daniel P. Berrangé" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Peter Maydell" <[email protected]>, "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Wainer Moschetta" <[email protected]>, "Markus Armbruster" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Alex Bennée" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Thursday, April 23, 2020 1:13:22 PM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 61 lines · Cleber Rosa

----- Original Message -----

Quoted 46 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik
Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>,
"QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo
Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 24 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these
guys.

Quoted 5 lines

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and
the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Hi Peter,

Quoted 8 lines
Quoted 5 lines

You hit the nail in the head here. We were affected indeed with our ability
to move some machines from one lab to another (across the country), but we're
actively working on it.

For x86, do we really need to be using custom runners ?

Hi Daniel,

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

Quoted 3 lines

My experiments were really around that point, I mean, confirming that we can grow the number of architectures/runners/jobs/configurations to provide a coverage equal or greater to what Peter already does.

So IIUC, you're saying that for x86 gating, the intention is to use shared runners in general.

Your current work that you say is blocked on access to x86 hardware, is just about demonstrating the concept of plugging in custom runners, while we wait for access to non-x86 hardware ?

Quoted 15 lines
Quoted 5 lines

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

They will continue to be able use the jobs and runners already defined in the .gitlab-ci.yml file. This work will only affect people pushing to the/a "staging" branch.

Quoted 4 lines

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Quoted 4 lines

If you're looking for symmetry between any PR and "merge time" jobs, the only solution is to allow any PR to access all the diverse set of non-shared machines we're hoping to have in the near future. This may be something we'll get to, but I doubt we can tackle it in the near future now.

It occurred to me that we could do this if we grant selective access to the Gitlab repos, to people who are official subsystem maintainers. GitLab has a concept of "protected branches", so you can control who is allowed to push changes on a per-branch granularity.

So, for example, in the main qemu.git, we could create branches for each subsystem tree eg

staging-block
staging-qapi
staging-crypto
staging-migration
....

and for each of these branches, we can grant access to relevant subsystem maintainer(s).

When they're ready to send a pull request to Peter, they can push their tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu project namespace, this branch can run CI using the private QEMU runners. The subsystem maintainer can thus see the full set of CI results across all platforms required by Gating, before Peter even gets the pull request.

So when Peter then looks at merging the pull request to master, the only he's likely to see are the non-deterministic bugs, or issues caused by semantic conflicts with other recently merged code.

It would even be possible to do the final merge into master entirely from GitLab, no need to go via email. When the source branch & target branch are within the same git repo, GitLab has the ability to run CI jobs against the resulting merge commit in a strict gating manner, before it hits master. They call this "Merge trains" in their documentation.

IOW, from Peter's POV, merging pull requests could be as simple as hitting the merge button in GitLab merge request UI. Everything wrt CI would be completely automated, and the subsystem maintainers would have the responsibility to dealing with merge conflicts & CI failures, which is more scalable for the person co-ordinating the merges into master.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#40 lore raw
Quoted 3 lines · Daniel P. Berrangé
Quoted 3 lines

----- Original Message -----

Quoted 2 lines

From: "Daniel P. Berrangé" <[email protected]>
To: "Cleber Rosa" <[email protected]>

[...]

Quoted 47 lines
Quoted 11 lines

Hi Daniel,

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

My experiments were really around that point, I mean, confirming that we can grow the number of architectures/runners/jobs/configurations to provide a coverage equal or greater to what Peter already does.

So IIUC, you're saying that for x86 gating, the intention is to use shared runners in general.

Your current work that you say is blocked on access to x86 hardware, is just about demonstrating the concept of plugging in custom runners, while we wait for access to non-x86 hardware ?

Quoted 15 lines
Quoted 5 lines

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

They will continue to be able use the jobs and runners already defined in the .gitlab-ci.yml file. This work will only affect people pushing to the/a "staging" branch.

Quoted 4 lines

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Quoted 4 lines

If you're looking for symmetry between any PR and "merge time" jobs, the only solution is to allow any PR to access all the diverse set of non-shared machines we're hoping to have in the near future. This may be something we'll get to, but I doubt we can tackle it in the near future now.

It occurred to me that we could do this if we grant selective access to the Gitlab repos, to people who are official subsystem maintainers. GitLab has a concept of "protected branches", so you can control who is allowed to push changes on a per-branch granularity.

So, for example, in the main qemu.git, we could create branches for each subsystem tree eg

Quoted 8 lines

staging-block
staging-qapi
staging-crypto
staging-migration
....

and for each of these branches, we can grant access to relevant subsystem maintainer(s).

The MAINTAINERS file could help us with that, we already have scripts to parse its sections.
Maintainers should keep it up-to-date, then the merge script would check, i.e.:

<newline> // section separator
--------------- // ignored
Trivial patches // description ignored
M: Michael Tokarev <[email protected]>
M: Laurent Vivier <[email protected]> // must match commit author
T: git git://git.corpit.ru/qemu.git trivial-patches
T: git https://github.com/vivier/qemu.git trivial-patches // must match MR source

Quoted 26 lines

When they're ready to send a pull request to Peter, they can push their tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu project namespace, this branch can run CI using the private QEMU runners. The subsystem maintainer can thus see the full set of CI results across all platforms required by Gating, before Peter even gets the pull request.

So when Peter then looks at merging the pull request to master, the only he's likely to see are the non-deterministic bugs, or issues caused by semantic conflicts with other recently merged code.

It would even be possible to do the final merge into master entirely from GitLab, no need to go via email. When the source branch & target branch are within the same git repo, GitLab has the ability to run CI jobs against the resulting merge commit in a strict gating manner, before it hits master. They call this "Merge trains" in their documentation.

IOW, from Peter's POV, merging pull requests could be as simple as hitting the merge button in GitLab merge request UI. Everything wrt CI would be completely automated, and the subsystem maintainers would have the responsibility to dealing with merge conflicts & CI failures, which is more scalable for the person co-ordinating the merges into master.

Regards,
Daniel

#41 lore raw
In reply to Daniel P. Berrangé
Quoted 100 lines · On Fri, 24 Apr 2020 10:30:23 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 99 lines · Cleber Rosa

----- Original Message -----

Quoted 78 lines

From: "Daniel P. Berrangé" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Peter Maydell" <[email protected]>, "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Wainer Moschetta" <[email protected]>, "Markus Armbruster" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Alex Bennée" <[email protected]>, "Eduardo Habkost" <[email protected]> Sent: Thursday, April 23, 2020 1:13:22 PM Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 64 lines · Cleber Rosa

----- Original Message -----

Quoted 48 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]> Sent: Tuesday, April 21, 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 26 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these
guys.

Quoted 5 lines

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and
the Red Hat runners.

Quoted 10 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Hi Peter,

Quoted 8 lines
Quoted 5 lines

You hit the nail in the head here. We were affected indeed with our ability
to move some machines from one lab to another (across the country), but we're
actively working on it.

For x86, do we really need to be using custom runners ?

Hi Daniel,

We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time.

Quoted 8 lines
Quoted 4 lines

My experiments were really around that point, I mean, confirming that we can grow the number of architectures/runners/jobs/configurations to provide a coverage equal or greater to what Peter already does.

So IIUC, you're saying that for x86 gating, the intention is to use shared runners in general.

No, I've said that whenever possible we could use containers and thus shared runners. For instance, testing QEMU running on the x86 CentOS 8 KVM is not something we could do with shared runners.

Quoted 4 lines

Your current work that you say is blocked on access to x86 hardware, is just about demonstrating the concept of plugging in custom runners, while we wait for access to non-x86 hardware ?

Short answer is no. The original scope and goal was to have the same or very similar jobs that Peter runs himself in his own machines. So it was/is not about just x86 hardware, but x86 that can run a couple of different OSs, and non-x86 hardware too. We're basically scaling down and changing the scope (for instance adding the s390 machine here) in an attempt to get this moving forward.

Quoted 38 lines
Quoted 15 lines
Quoted 5 lines

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

They will continue to be able use the jobs and runners already defined in the .gitlab-ci.yml file. This work will only affect people pushing to the/a "staging" branch.

Quoted 4 lines

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Quoted 5 lines

If you're looking for symmetry between any PR and "merge time" jobs, the only solution is to allow any PR to access all the diverse set of non-shared machines we're hoping to have in the near future. This may be something we'll get to, but I doubt we can tackle it in the near future now.

It occurred to me that we could do this if we grant selective access to the Gitlab repos, to people who are official subsystem maintainers. GitLab has a concept of "protected branches", so you can control who is allowed to push changes on a per-branch granularity.

So, for example, in the main qemu.git, we could create branches for each subsystem tree eg

staging-block
staging-qapi
staging-crypto
staging-migration
....

and for each of these branches, we can grant access to relevant subsystem maintainer(s).

Quoted 7 lines

When they're ready to send a pull request to Peter, they can push their tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu project namespace, this branch can run CI using the private QEMU runners. The subsystem maintainer can thus see the full set of CI results across all platforms required by Gating, before Peter even gets the pull request.

Sure, this is actually an extrapolation/extension of what we're proposing to do here with the unique "staging" branch. I see no issues at all to have more than one (one per subsystem/maintainer) staging branches.

Quoted 11 lines

So when Peter then looks at merging the pull request to master, the only he's likely to see are the non-deterministic bugs, or issues caused by semantic conflicts with other recently merged code.

It would even be possible to do the final merge into master entirely from GitLab, no need to go via email. When the source branch & target branch are within the same git repo, GitLab has the ability to run CI jobs against the resulting merge commit in a strict gating manner, before it hits master. They call this "Merge trains" in their documentation.

Quoted 7 lines

IOW, from Peter's POV, merging pull requests could be as simple as hitting the merge button in GitLab merge request UI. Everything wrt CI would be completely automated, and the subsystem maintainers would have the responsibility to dealing with merge conflicts & CI failures, which is more scalable for the person co-ordinating the merges into master.

This is very much aligned to some previous discussions, I believe, at the RFC thread. But for practical purposes, the general direction was to scale down to the bare minimum to replicate Peter's setup and workflow, and then move from there to possibly something very similar to what you're describing here.

Quoted 3 lines

Regards,
Daniel

Thanks a *whole lot* for the feedback Daniel!
- Cleber.

#42 lore raw
In reply to Daniel P. Berrangé
Quoted 72 lines · Daniel P. Berrangé
Quoted 50 lines · Cleber Rosa

----- Original Message -----

Quoted 39 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 5 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Quoted 8 lines

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner. Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error.

A non-reproducible runner can not be used as gating, because if they fail it is not acceptable to lock the project development process.

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

[*] Specific hardware that is not easily available.

- Alistair at last KVM forum talked about a RISCV board (to test host TCG)
- Aleksandar said at last KVM forum Wavecomp could plug a CI20 MIPS (to test host TCG)
- Lemote seems interested to setup some Loongson MIPSr6 board (to test interaction with KVM)

[*] To run code requiring accepting License Agreements

[*] To run non Free / Open Source code

Owner of these runners take the responsibility to provide enough time/information about reported bugs, or to debug them themselves.

Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected.

#43 lore raw
Quoted 73 lines · Philippe Mathieu-Daudé
Quoted 72 lines · Daniel P. Berrangé
Quoted 50 lines · Cleber Rosa

----- Original Message -----

Quoted 39 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 5 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Quoted 8 lines
Quoted 8 lines

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

Not just ^that, you also want custom VM runners to run integration tests, e.g. in libvirt, we'd have to put systemd and a lof of other cruft into the container to be able to run the tests at which point you must ask yourself, whyt not go with a VM instead in which case we're limited in terms of infrastructure...

Quoted 6 lines

Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner.
Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error.

(Thanks for bringing ^this point up Philippe)

...However, what we've been actively working on in libvirt is to extend the lcitool we have (which can spawn local test VMs) to the point where we're able to generate machines that would be the reproducible. Right now I'm playing with cloud-init integration with lcitool (patches coming soon) that would allow us to use the same machines locally as we'd want to have in, say, OpenStack and share them as compressed images, so even when updated/managed by lcitool locally, you'd get the same environment.

Regards,

Signature · 1 line
Erik Skultety
#44 lore raw
Quoted 95 lines · On Fri, 24 Apr 2020 08:57:46 +0200 Erik Skultety <[email protected]> wrote:
Quoted 83 lines · Philippe Mathieu-Daudé
Quoted 82 lines · Daniel P. Berrangé
Quoted 59 lines · Cleber Rosa

----- Original Message -----

Quoted 47 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]> Sent: Tuesday, April 21, 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 25 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

Quoted 5 lines

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 10 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 5 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Quoted 9 lines
Quoted 9 lines

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

Quoted 6 lines

Not just ^that, you also want custom VM runners to run integration tests, e.g. in libvirt, we'd have to put systemd and a lof of other cruft into the container to be able to run the tests at which point you must ask yourself, whyt not go with a VM instead in which case we're limited in terms of infrastructure...

I'm completely in agreement that a lot of the jobs will be suitable to be run on containers, but like you exemplified here Erik, we must take into account the ones that won't be suitable.

For instance, a very real use case is testing KVM on bare metal. One could argue that "QEMU running on a container making use of KVM would suffice". It may be true, it may not. But even that won't be possible Today on a CentOS/RHEL 8 machine, because gitlab-runner knows nothing about podman, so full blown x86 physical boxes may be the "cheaper" and more practical solution, at least initially. Trying to use other architectures would surely have similar issues.

Quoted 18 lines
Quoted 6 lines

Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner.
Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error.

(Thanks for bringing ^this point up Philippe)

...However, what we've been actively working on in libvirt is to extend the lcitool we have (which can spawn local test VMs) to the point where we're able to generate machines that would be the reproducible. Right now I'm playing with cloud-init integration with lcitool (patches coming soon) that would allow us to use the same machines locally as we'd want to have in, say, OpenStack and share them as compressed images, so even when updated/managed by lcitool locally, you'd get the same environment.

Quoted 2 lines

Regards,

This is great, and it aligns with my previous point that reproducibility it's not *just* about the hardware, but about diligently documenting and automating the environments, be them mundane or super specialized. And IMO that should score some points when it comes to being promoted/demoted as a Gating machine/job.

Thanks for the feedback Erik!
- Cleber.

#45 lore raw
Quoted 21 lines · Cleber Rosa
Quoted 10 lines · On Fri, 24 Apr 2020 08:57:46 +0200 Erik Skultety <[email protected]> wrote:
Quoted 5 lines · Philippe Mathieu-Daudé

Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner.
Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error.

(Thanks for bringing ^this point up Philippe)

Quoted 8 lines

...However, what we've been actively working on in libvirt is to extend the lcitool we have (which can spawn local test VMs) to the point where we're able to generate machines that would be the reproducible. Right now I'm playing with cloud-init integration with lcitool (patches coming soon) that would allow us to use the same machines locally as we'd want to have in, say, OpenStack and share them as compressed images, so even when updated/managed by lcitool locally, you'd get the same environment.

Quoted 5 lines

This is great, and it aligns with my previous point that reproducibility it's not *just* about the hardware, but about diligently documenting and automating the environments, be them mundane or super specialized. And IMO that should score some points when it comes to being promoted/demoted as a Gating machine/job.

I think there's room to extend and rework lcitool so that it can be used for QEMU CI as well, and we should definitely look into that.

Right now it only really covers VMs and containers, but there's already one situation (FreeBSD) where the expectation is that you'd import an existing VM image and then apply CI-related customizations on top, so it's not too much of a stretch to imagine doing the same for a bare metal machine. We use Ansible, so as long as we can connect to the machine via ssh we're pretty much good to go.

Installation of VMs we already perform in an unattended fashion using preseed/kickstart, and it should be relatively straighforward to adapt those configurations to also work on real hardware. This way we'd both be able to rely on having a sane OS as the base, and relieve the administrator from the duty of manually installing the machines.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#46 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 88 lines · On Thu, 23 Apr 2020 23:28:21 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 78 lines · Daniel P. Berrangé
Quoted 55 lines · Cleber Rosa

----- Original Message -----

Quoted 43 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]> Sent: Tuesday, April 21, 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 10 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 5 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs.

Quoted 8 lines

Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason.

Quoted 3 lines

Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner.

I also believe that having jobs that can be reproducible is key, but I differ when it comes to believing that the hardware *alone* should define if a job is gating or not.

My point is that even with easily accessible systems and software, different jobs can easily yield different results, because of how the underlying system is configured. Sure, containers, but again, we have to consider non container usage too.

In the RFC I tried to gather feedback on a plan to promote and demote jobs from a gating status. IMO, most jobs would begin their lives being non-gating, and would prove both their stability and their mantainer's responsiveness. Even when such jobs are already gating, it should be trivial to simply demote a gating job because of (and not limited to) any of the following reasons:

* job fails in a non-reproducible way
* hardware is unresponsive and takes too long to produce results
* maintainer is MIA

Some or all of the gating runners could also pick up jobs sent to a branch other than "staging", say, a branch called "reproducer". That branch could be writable by maintainers that need to reproduce a given failure.

Quoted 3 lines

Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error.

One of the goals of the patches you'll find on this series is to propose (I would say *require*) that new jobs that require new hardware (even easily accessible systems such as x86) should provide easy to run scripts to recreate those environments. This is inline with my previous point that it's not enough to just have the same hardware.

Quoted 3 lines

A non-reproducible runner can not be used as gating, because if they fail it is not acceptable to lock the project development process.

Other people may be more familiar with this, but I do remember projects such as OpenStack deferring test of hardware/software combinations to other entities. One specific party won't be able to reproduce all configurations unless it's decided to be kept really small. In my opinion, it's better to face it and acknowledge that fact, and have plans to be put to action in the exceptional cases where the environment to reproduce a test is now unavailable.

Quoted 4 lines

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on. If their system is not gating, they lose that and may find breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status.

For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it?

Quoted 9 lines

[*] Specific hardware that is not easily available.

- Alistair at last KVM forum talked about a RISCV board (to test host TCG)
- Aleksandar said at last KVM forum Wavecomp could plug a CI20 MIPS (to test host TCG)
- Lemote seems interested to setup some Loongson MIPSr6 board (to test interaction with KVM)

Quoted 8 lines

[*] To run code requiring accepting License Agreements

[*] To run non Free / Open Source code

Owner of these runners take the responsibility to provide enough time/information about reported bugs, or to debug them themselves.

I do think that the owner of such runners may *not* want to have them with Gating jobs, but I don't think the opposite should be the case, because I find it very hard to define, without some prejudice, what "easily available runner" means.

Quoted 6 lines

Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected.

That's where I'm trying to help. I built and tested the gitlab-runner for a number of non-supported environments, and I expect to build further on that (say contributing code or feedback back to GitLab so they become official builds?).

Cheers,
- Cleber.

#47 lore raw
Quoted 2 lines · Cleber Rosa

On Thu, 23 Apr 2020 23:28:21 +0200
Philippe Mathieu-Daudé <[email protected]> wrote:

[...]

Quoted 12 lines
Quoted 3 lines

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on. If their system is not gating, they lose that and may find breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status.

Quoted 4 lines

For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it?

I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"?

[...]

Quoted 10 lines
Quoted 5 lines

Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected.

That's where I'm trying to help. I built and tested the gitlab-runner for a number of non-supported environments, and I expect to build further on that (say contributing code or feedback back to GitLab so they become official builds?).

Good luck with that, it took more that 2 years to GitLab to officially support AMD64:
https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725

Hopefully the first non-x86 user was the hardest one who had to do all the bad work, and next architecture might get supported quicker...

Quoted 4 lines

Cheers,
- Cleber.

#48 lore raw
Quoted 17 lines · On Mon, 27 Apr 2020 12:51:36 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 2 lines · Cleber Rosa

On Thu, 23 Apr 2020 23:28:21 +0200
Philippe Mathieu-Daudé <[email protected]> wrote:

[...]

Quoted 13 lines
Quoted 3 lines

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on.
If their system is not gating, they lose that and may find
breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status.

Quoted 8 lines
Quoted 4 lines

For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it?

I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"?

What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware?

Quoted 7 lines

[...]

Quoted 6 lines
Quoted 5 lines

Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected.

Quoted 9 lines
Quoted 4 lines

That's where I'm trying to help. I built and tested the gitlab-runner for a number of non-supported environments, and I expect to build further on that (say contributing code or feedback back to GitLab so they become official builds?).

Good luck with that, it took more that 2 years to GitLab to officially support AMD64:
https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725

You mean aarch64, sure. I'm not holding my breath, because we can always have our own binaries/ports (or other executors such as ssh) but I'm optimistic...

Quoted 3 lines

Hopefully the first non-x86 user was the hardest one who had to do all the bad work, and next architecture might get supported quicker...

... and this point is one of the reasons. The other is competition from Travis-CI (and others).

Cheers,
- Cleber.

#49 lore raw
Quoted 25 lines · Cleber Rosa
Quoted 22 lines · On Mon, 27 Apr 2020 12:51:36 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 2 lines · Cleber Rosa

On Thu, 23 Apr 2020 23:28:21 +0200
Philippe Mathieu-Daudé <[email protected]> wrote:

[...]

Quoted 17 lines
Quoted 3 lines

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on.
If their system is not gating, they lose that and may find
breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status.

For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it?

Quoted 8 lines
Quoted 3 lines

I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"?

What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware?

My view on this is if someone merged code in mainstream QEMU and maintains it, and if it is not easy to reproduce the setup (for a bug reported by a CI script), then it is the responsibility of the maintainer to resolve it. Either by providing particular access to the hardware, or be ready to spend a long debugging session over email and multiple time zones.

If it is not possible, then this specific code/setup can not claim for gating CI, and eventually mainstream isn't the best place for it.

Quoted 1 line
Quoted 1 line

[...]

#50 lore raw
Quoted 7 lines · On Mon, 27 Apr 2020 16:41:38 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 5 lines · Cleber Rosa

What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware?

Quoted 7 lines

My view on this is if someone merged code in mainstream QEMU and maintains it, and if it is not easy to reproduce the setup (for a bug reported by a CI script), then it is the responsibility of the maintainer to resolve it. Either by providing particular access to the hardware, or be ready to spend a long debugging session over email and multiple time zones.

Right, the "easy to reproduce" has a lot to with access to hardware, and a lot to do with access to the same or reproducible setup. And yes, if I maintain platform/job "foobar" that was once upgraded to gating status, has since then fallen behind and doesn't allow users to easily reproduce it, it all falls unto the maintainer to resolve issues.

I'd even say that people having access to identical hardware could proactively challenge a given job status as gating if they fail to reproduce it with the provided documentation/scripts.

Quoted 5 lines

If it is not possible, then this specific code/setup can not claim for gating CI, and eventually mainstream isn't the best place for it.

Quoted 1 line
Quoted 1 line

[...]

IIUC, we're in agreement. :)

Thanks,
- CLeber.

#51 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 25 lines · Cleber Rosa
Quoted 22 lines · On Mon, 27 Apr 2020 12:51:36 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 2 lines · Cleber Rosa

On Thu, 23 Apr 2020 23:28:21 +0200
Philippe Mathieu-Daudé <[email protected]> wrote:

[...]

Quoted 17 lines
Quoted 3 lines

In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status.

Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on.
If their system is not gating, they lose that and may find
breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status.

For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it?

Quoted 8 lines
Quoted 3 lines

I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"?

What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware?

Quoted 8 lines

My view on this is if someone merged code in mainstream QEMU and maintains it, and if it is not easy to reproduce the setup (for a bug reported by a CI script), then it is the responsibility of the maintainer to resolve it. Either by providing particular access to the hardware, or be ready to spend a long debugging session over email and multiple time zones.

If it is not possible, then this specific code/setup can not claim for gating CI, and eventually mainstream isn't the best place for it.

I'd caution to be wary about using gating CI as a big stick for hitting contributors with. The more rules we put in place whicih contributors have to follow before their work gets accepted for merge, the less likely someone is to have a positive experiance contributing to the project, or even be willing to try. This view of gating CI requirements was a negative aspect of contributing to the OpenStack project, which drove people away. There was pushback against contributing work because it lacked CI, but there was often no viable way for to actually provide CI in a feasible timeframe, especially for stuff only testable in physical hardware and not VMs. Even if you work for a big company, it doesn't make it easy to magic up money to spend on hardware & hosting to provide CI, as corporate beaurcracy & priorities will get in your way.

I'd really encourage the more nuanced approach of thinking in terms of tiered support levels:

- Tier 1: features that we have gating CI tests for. Will always work.

- Tier 2: features that we have non-gating CI test for. Should work at time of release, but may be broken for periods in git master.

- Tier 3: features that we don't have CI tests for. Compile tested only, relying on end user manual testing, so may or may not work at any time or in release.

Obviously tier 1 is the gold standard that we would like everything to achieve but we'll never achieve that reality unless we cull 90% of QEMU's code. I don't think that's in the best interests of our users, because clearly stuff in Tier 2 and Tier 3 is still useful for a large portion of our end users - not least because Tier 3 is the level everything is at right now in QEMU unless using a downstream vendor's packages.

The tier levels and CI are largely around setting reasonable quality expectations. Right now we often have a problem that poeople want to re-factor code but are afraid of breaking existing functionality that guests rely on. This causes delays in merging code or causes people to not even attempt the refactoring in the first place. This harms our forward progress in QEMU.

With gating CI, we are declaring that contributors should feel free to refactor anything as long as it passes gating CI. IOW, contributors only have to care about Tier 1 features continuing to work. It would be nice if refactoring does not breaks stuff in Tier 2 / 3, but if it does, then that is acceptable collatoral damage. We would not block the merge on stuff that is Tier 2 / 3.

Based on what I experianced in OpenStack the other big challenge is deciding when something can be promoted from Tier 2 to Tier 1. They had the official gating CI (for Tier 1) being maintained by the core project infrastructure team. Any CI provided by third party companies was non-gating (Tier 2) (at least in the time I was involved) because they did not want code merge blocked on ability to communicate with third party companies who were often hard to contact when CI broke. So the only real path from Tier 2 to Tier 1 was to give the project direct access to the CI hardware, instead of having the providing company self-manage it.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#52 lore raw
In reply to Cleber Rosa
Quoted 52 lines · Cleber Rosa

----- Original Message -----

Quoted 39 lines

From: "Peter Maydell" <[email protected]>
To: "Markus Armbruster" <[email protected]>
Cc: "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Beraldo Leal" <[email protected]>, "Erik Skultety" <[email protected]>, "Alex Bennée" <[email protected]>, "Wainer Moschetta" <[email protected]>, "QEMU Developers" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Cleber Rosa" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Eduardo Habkost" <[email protected]>
Sent: Tuesday, April 21, 2020 8:53:49 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Peter Maydell

I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

Quoted 8 lines

* Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys.

A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners.

Quoted 9 lines

So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ?

Quoted 9 lines

Hi Peter,

You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it.

From now on, I'll give you an update every time a significant event occurs on our side.

Hi all,

It took a while, but I finally have some updates, and they're pretty good news.

Red Hat is sponsoring 3 x86_64 machines that will act as runners for the QEMU CI, and together with QEMU's own s390 and aarch64 machines (1 each), we have enough hardware now for a reasonable build and test coverage.

The s390 and aarch64 machines are already up and running, and the x86_64 machines are being racked up, but should be up and running in the next few days. I'm working on an updated version of this series that takes into account this new scenario, and some fixes and improvements.

And as a reminder, if you (individual or organization) would like to sponsor hardware or people to expand the QEMU build and test coverage, please reach out to us.

Thanks,
- Cleber.

Quoted 7 lines
Quoted 4 lines

thanks
-- PMM

Thanks for checking in!
- Cleber.

#53 lore raw
In reply to Cleber Rosa
Quoted 35 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
tests/docker: add CentOS 8 Dockerfile
tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts

Quoted 7 lines
 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++

FYI, the contrib/ directory is generally a place for arbitrary / adhoc but interesting user contributed files/sources that are not officially supported deliverables of the project.

IOW, this is not a good home for the official CI scripts.

We already have a .gitlab-ci.d/ directory that looks like it would be good for this. Or if that's not suitable, then scripts/ci/ is a second choice.

Quoted 11 lines
 docs/devel/testing.rst                        | 142 ++++++++++++
 tests/docker/dockerfiles/centos8.docker       |  32 +++
 .../dockerfiles/debian-win32-cross.docker     |   2 +-
 10 files changed, 751 insertions(+), 4 deletions(-)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
 create mode 100644 tests/docker/dockerfiles/centos8.docker

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#54 lore raw

----- Original Message -----

Quoted 54 lines

From: "Daniel P. Berrangé" <[email protected]>
To: "Cleber Rosa" <[email protected]>
Cc: "Alex Bennée" <[email protected]>, "Peter Maydell" <[email protected]>, [email protected], "Fam Zheng" <[email protected]>, "Thomas Huth" <[email protected]>, "Eduardo Habkost" <[email protected]>, "Erik Skultety" <[email protected]>, "Wainer Moschetta" <[email protected]>, "Wainer dos Santos Moschetta" <[email protected]>, "Willian Rampazzo" <[email protected]>, "Philippe Mathieu-Daudé" <[email protected]>, "Beraldo Leal" <[email protected]>
Sent: Monday, March 16, 2020 8:38:07 AM
Subject: Re: [PATCH 0/5] QEMU Gating CI

Quoted 35 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
tests/docker: add CentOS 8 Dockerfile
tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts

Quoted 7 lines
 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
Quoted 6 lines

FYI, the contrib/ directory is generally a place for arbitrary / adhoc but interesting user contributed files/sources that are not officially supported deliverables of the project.

IOW, this is not a good home for the official CI scripts.

Good point, reason is that I wasn't/ain't sure this script is going to be "official" and really that useful. I'm happy to move it somewhere else though.

Quoted 4 lines

We already have a .gitlab-ci.d/ directory that looks like it would be good for this. Or if that's not suitable, then scripts/ci/ is a second choice.

Ack.

Thanks!
- Cleber.

Quoted 22 lines
Quoted 11 lines
 docs/devel/testing.rst                        | 142 ++++++++++++
 tests/docker/dockerfiles/centos8.docker       |  32 +++
 .../dockerfiles/debian-win32-cross.docker     |   2 +-
 10 files changed, 751 insertions(+), 4 deletions(-)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
 create mode 100644 tests/docker/dockerfiles/centos8.docker

Regards,
Daniel
--

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
|:|
|: https://libvirt.org         -o-            https://fstop138.berrange.com
|:|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
|:|
#55 lore raw
In reply to Daniel P. Berrangé
Quoted 48 lines · Daniel P. Berrangé
Quoted 35 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
tests/docker: add CentOS 8 Dockerfile
tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts

Quoted 7 lines
 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++

FYI, the contrib/ directory is generally a place for arbitrary / adhoc but interesting user contributed files/sources that are not officially supported deliverables of the project.

Quoted 5 lines

IOW, this is not a good home for the official CI scripts.

We already have a .gitlab-ci.d/ directory that looks like it would be good for this. Or if that's not suitable, then scripts/ci/ is a second choice.

I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be a little hidden.

Signature · 1 line
Alex Bennée
#56 lore raw
Quoted 52 lines · Alex Bennée
Quoted 48 lines · Daniel P. Berrangé
Quoted 35 lines · Cleber Rosa

The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
(https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs make check
- Added "-jnproc" and "-jsysctl -n hw.ncpu" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
tests/docker: add CentOS 8 Dockerfile
tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts

Quoted 7 lines
 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++

FYI, the contrib/ directory is generally a place for arbitrary / adhoc but interesting user contributed files/sources that are not officially supported deliverables of the project.

Quoted 9 lines
Quoted 5 lines

IOW, this is not a good home for the official CI scripts.

We already have a .gitlab-ci.d/ directory that looks like it would be good for this. Or if that's not suitable, then scripts/ci/ is a second choice.

I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be a little hidden.

I vote for scripts/ci/ or scripts/gitlab/ too. With a little preference to scripts/ci/.

Aleksandar

Quoted 3 lines

--
Alex Bennée