gitlab-ci: Maintain a public runner, allow manual pipeline
cover gitlab-ci: Maintain a public runner, allow manual pipeline
Yet another attempt to make GitLab-CI inclusive to the all users from the QEMU community.
We allow manual pipeline for some users, letting default to all jobs started.
Also I step in to maintain in my work time the public runner based on Fedora that we use for the X86 32-bit builds, because I don't want it to bitrot, as it is important for my daily work.
Regards,
Phil.
Philippe Mathieu-Daudé (2):
gitlab-ci: Step in to maintain the fedora-i386-cross runner
gitlab-ci: Introduce allow_skipping_job_template
.gitlab-ci.d/crossbuilds.yml | 19 +++++++++++++++++++ .gitlab-ci.yml | 18 ++++++++++++++++++ 2 files changed, 37 insertions(+)
Signature · 1 line
2.26.2
RFC PATCH 1/2 gitlab-ci: Step in to maintain the fedora-i386-cross runner
As I am interested in using the runner based on the fedora-i386-cross docker image, add a smoke test job to be sure this image is usable, and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so other developers can contact me in case there is a problem with the public runner based on this image.
We will eventually send a patch to make script/checkpatch.pl parse the Gitlab YAML jobs to understand the JOB_MAINTAINER_NAME and JOB_MAINTAINER_EMAIL variables.
--- .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
.gitlab-ci.d/crossbuilds.yml +15 -0
@@ -39,6 +39,13 @@ ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system - make -j$(expr $(nproc) + 1) all check-build +.cross_sanity_check_job: + stage: build + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + timeout: 3m + script: + - /bin/true + cross-armel-system: extends: .cross_system_build_job variables:
@@ -69,6 +76,14 @@ cross-arm64-user: variables: IMAGE: debian-arm64-cross +cross-i386-sanity-check: + extends: .cross_sanity_check_job + variables: + JOB_MAINTAINER_NAME: "Philippe Mathieu-Daudé" + JOB_MAINTAINER_EMAIL: [email protected] + GIT_SUBMODULE_STRATEGY: none + IMAGE: fedora-i386-cross + cross-mips-system: extends: .cross_system_build_job variables:
Signature · 1 line
2.26.2
Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner
Quoted 20 lines · Philippe Mathieu-Daudé
As I am interested in using the runner based on the fedora-i386-cross docker image, add a smoke test job to be sure this image is usable, and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so other developers can contact me in case there is a problem with the public runner based on this image.
We will eventually send a patch to make
script/checkpatch.plparse the Gitlab YAML jobs to understand theJOB_MAINTAINER_NAMEandJOB_MAINTAINER_EMAILvariables.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+).gitlab-ci.d/crossbuilds.yml +0
-0
.gitlab-ci.d/crossbuilds.yml line 47
Quoted 9 lines
.gitlab-ci.d/crossbuilds.yml +6
-0@@ -39,6 +39,13 @@ ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system - make -j$(expr $(nproc) + 1) all check-build +.cross_sanity_check_job: + stage: build + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + timeout: 3m + script: + - /bin/true
I guess it makes more sense if I respin my patch here instead:
https://www.mail-archive.com/[email protected]/msg762919.html
?
Thomas
Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner
Quoted 33 lines · Thomas Huth
Quoted 20 lines · Philippe Mathieu-Daudé
As I am interested in using the runner based on the fedora-i386-cross docker image, add a smoke test job to be sure this image is usable, and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so other developers can contact me in case there is a problem with the public runner based on this image.
We will eventually send a patch to make
script/checkpatch.plparse the Gitlab YAML jobs to understand theJOB_MAINTAINER_NAMEandJOB_MAINTAINER_EMAILvariables.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+).gitlab-ci.d/crossbuilds.yml +0
-0.gitlab-ci.d/crossbuilds.yml line 47
Quoted 9 lines
.gitlab-ci.d/crossbuilds.yml +6
-0@@ -39,6 +39,13 @@ ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system - make -j$(expr $(nproc) + 1) all check-build +.cross_sanity_check_job: + stage: build + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + timeout: 3m + script: + - /bin/trueI guess it makes more sense if I respin my patch here instead:
I think these are different. The idea of testing /bin/true is to be able to make the distinction between a broken runner and a broken test.
Quoted 3 lines
Thomas
Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 33 lines · Thomas Huth
Quoted 20 lines · Philippe Mathieu-Daudé
As I am interested in using the runner based on the fedora-i386-cross docker image, add a smoke test job to be sure this image is usable, and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so other developers can contact me in case there is a problem with the public runner based on this image.
We will eventually send a patch to make
script/checkpatch.plparse the Gitlab YAML jobs to understand theJOB_MAINTAINER_NAMEandJOB_MAINTAINER_EMAILvariables.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+).gitlab-ci.d/crossbuilds.yml +0
-0.gitlab-ci.d/crossbuilds.yml line 47
Quoted 9 lines
.gitlab-ci.d/crossbuilds.yml +6
-0@@ -39,6 +39,13 @@ ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system - make -j$(expr $(nproc) + 1) all check-build +.cross_sanity_check_job: + stage: build + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + timeout: 3m + script: + - /bin/trueI guess it makes more sense if I respin my patch here instead:
Quoted 7 lines
I think these are different. The idea of testing
/bin/trueis to be able to make the distinction between a broken runner and a broken test.
I don't see any point in this job. /bin/true will always succeeed if we've successfully created the container image if the container build job.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 2/2 gitlab-ci: Introduce allow_skipping_job_template
In reply to Philippe Mathieu-DaudéIt might be overkill (and frustrating) to rerun all the jobs when iterating over a single code unit. As some users are interested in only rerunning a specific set of jobs, allow them to use a "manual" profile, when jobs are created in the pipeline but not started. It is up to the user to manually start the jobs of interest.
To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL in the project UI, following [1].
Once a repository is using the manual workflow, the user can still automatically start the jobs by explicitly set the QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
Example while pushing a branch/tag from command line (see [2]):
$ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
myrepo mybranch[1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
[2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
--- .gitlab-ci.d/crossbuilds.yml | 4 ++++ .gitlab-ci.yml | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+)
.gitlab-ci.d/crossbuilds.yml +4 -0
@@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m
@@ -18,6 +19,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $ACCEL_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m
@@ -30,6 +32,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_user_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
@@ -40,6 +43,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_sanity_check_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 3m
.gitlab-ci.yml +18 -0
@@ -13,7 +13,21 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' +.allow_skipping_job_template: &allow_skipping_job_definition + rules: + # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its + # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW + # variable. + - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null + when: manual + allow_failure: true + - when: always + +.allow_skipping_job: + <<: *allow_skipping_job_definition + .native_build_job_template: &native_build_job_definition + <<: *allow_skipping_job_definition stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script:
@@ -35,6 +49,7 @@ include: fi .native_test_job_template: &native_test_job_definition + <<: *allow_skipping_job_definition stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
@@ -43,6 +58,7 @@ include: - make $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition + <<: *allow_skipping_job_definition cache: key: "${CI_JOB_NAME}-cache" paths:
@@ -536,6 +552,7 @@ check-dco: GIT_DEPTH: 1000 build-libvhost-user: + <<: *allow_skipping_job_definition stage: build image: $CI_REGISTRY_IMAGE/qemu/fedora:latest before_script:
@@ -547,6 +564,7 @@ build-libvhost-user: - ninja pages: + <<: *allow_skipping_job_definition image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest stage: test needs:
Signature · 1 line
2.26.2
Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template
Quoted 55 lines · Philippe Mathieu-Daudé
It might be overkill (and frustrating) to rerun all the jobs when iterating over a single code unit. As some users are interested in only rerunning a specific set of jobs, allow them to use a "manual" profile, when jobs are created in the pipeline but not started. It is up to the user to manually start the jobs of interest.
To switch a repository to manual, set
QEMU_DEFAULT_WORKFLOW_MANUALin the project UI, following [1].Once a repository is using the manual workflow, the user can still automatically start the jobs by explicitly set the
QEMU_IGNORE_DEFAULT_WORKFLOWwhich disable the manual mode.Example while pushing a branch/tag from command line (see [2]):
$ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \ myrepo mybranch[1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
[2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicdSigned-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 4 ++++ .gitlab-ci.yml | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+).gitlab-ci.d/crossbuilds.yml +3
-0@@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m@@ -18,6 +19,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $ACCEL_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m@@ -30,6 +32,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_user_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
.gitlab-ci.d/crossbuilds.yml line 46
Quoted 4 lines
.gitlab-ci.d/crossbuilds.yml +1
-0@@ -40,6 +43,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_sanity_check_job: + extends: .allow_skipping_job
So "extends" is possible ....
Quoted 13 lines
.gitlab-ci.d/crossbuilds.yml +0
-0stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 3m.gitlab-ci.yml +2
-0@@ -13,7 +13,21 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' +.allow_skipping_job_template: &allow_skipping_job_definition + rules:
.gitlab-ci.yml include: · line 27
Quoted 10 lines
.gitlab-ci.yml +10
-0+ # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its + # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW + # variable. + - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null + when: manual + allow_failure: true + - when: always + +.allow_skipping_job: + <<: *allow_skipping_job_definition
... before the anchestor has been defined? That's nice!
I'm basically fine with this patch, but can you please also add some sentence how to use this to docs/devel/testing.rst ?
Thanks,
Thomas
Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template
Quoted 74 lines · Thomas Huth
Quoted 55 lines · Philippe Mathieu-Daudé
It might be overkill (and frustrating) to rerun all the jobs when iterating over a single code unit. As some users are interested in only rerunning a specific set of jobs, allow them to use a "manual" profile, when jobs are created in the pipeline but not started. It is up to the user to manually start the jobs of interest.
To switch a repository to manual, set
QEMU_DEFAULT_WORKFLOW_MANUALin the project UI, following [1].Once a repository is using the manual workflow, the user can still automatically start the jobs by explicitly set the
QEMU_IGNORE_DEFAULT_WORKFLOWwhich disable the manual mode.Example while pushing a branch/tag from command line (see [2]):
$ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \ myrepo mybranch[1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
[2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicdSigned-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 4 ++++ .gitlab-ci.yml | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+).gitlab-ci.d/crossbuilds.yml +3
-0@@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m@@ -18,6 +19,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $ACCEL_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m@@ -30,6 +32,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_user_build_job: + extends: .allow_skipping_job stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:.gitlab-ci.d/crossbuilds.yml line 46
Quoted 4 lines
.gitlab-ci.d/crossbuilds.yml +1
-0@@ -40,6 +43,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_sanity_check_job: + extends: .allow_skipping_jobSo "extends" is possible ....
Quoted 11 lines
.gitlab-ci.d/crossbuilds.yml +0
-0stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 3m.gitlab-ci.yml +0
-0@@ -13,7 +13,21 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml'
Quoted 14 lines
.gitlab-ci.yml include: · line 27
Quoted 12 lines
.gitlab-ci.yml +12
-0+.allow_skipping_job_template: &allow_skipping_job_definition + rules: + # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its + # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW + # variable. + - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null + when: manual + allow_failure: true + - when: always + +.allow_skipping_job: + <<: *allow_skipping_job_definition... before the anchestor has been defined? That's nice!
'extends' is indeed more powerful and less hackish.
Quoted 2 lines
I'm basically fine with this patch, but can you please also add some sentence how to use this to
docs/devel/testing.rst?
Thanks, I later discovered this isn't complete, because if a children job is in automatic state and a parent (dependency) is in manual, there is some "dead lock": the pipeline waits until manual job is started.
To be complete, I have to ensure if a job is in manual state, all its dependencies also are.
Then we are back to 'extends' VS 'anchors', as this doesn't work with anchors, I have to convert all jobs to 'extends'. Good news is I already did it, I simply need to rebase this series:
https://www.mail-archive.com/[email protected]/msg758331.html
Thanks for reviewing,
Phil.