gitlab-ci: Fix EDK2 & OpenSBI jobs
cover gitlab-ci: Fix EDK2 & OpenSBI jobs
We had rules to restrict building the EDK2 & OpenSBI firmwares on GitLab CI. Unfortunately commit 922febe2 ("Move edk2 and opensbi YAML files to .gitlab-ci.d folder") made these rules ineffective. Fix that. In particular avoid the EDK2 job burn all free minutes for our fork users.
Philippe Mathieu-Daudé (2):
gitlab-ci: Avoid running the OpenSBI job when not necessary
gitlab-ci: Avoid running the EDK2 job when not necessary
.gitlab-ci.d/edk2.yml | 11 ++++------- .gitlab-ci.d/opensbi.yml | 13 ------------- 2 files changed, 4 insertions(+), 20 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/2 gitlab-ci: Avoid running the OpenSBI job when not necessary
The OpenSBI jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [*]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
In commit 922febe2af we moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.
We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
As this job doesn't take much (less than 1min), run it by default.
[*] https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
--- .gitlab-ci.d/opensbi.yml | 13 ------------- 1 file changed, 13 deletions(-)
.gitlab-ci.d/opensbi.yml +0 -13
@@ -1,10 +1,5 @@ docker-opensbi: stage: containers - rules: # Only run this job when the Dockerfile is modified - - changes: - - .gitlab-ci.d/opensbi.yml - - .gitlab-ci.d/opensbi/Dockerfile - when: always image: docker:19.03.1 services: - docker:19.03.1-dind
@@ -25,14 +20,6 @@ docker-opensbi: build-opensbi: stage: build - rules: # Only run this job when ... - - changes: # ... roms/opensbi/ is modified (submodule updated) - - roms/opensbi/* - when: always - - if: '$CI_COMMIT_REF_NAME =~ /^opensbi/' # or the branch/tag starts with 'opensbi' - when: always - - if: '$CI_COMMIT_MESSAGE =~ /opensbi/i' # or last commit description contains 'OpenSBI' - when: always artifacts: paths: # 'artifacts.zip' will contains the following files: - pc-bios/opensbi-riscv32-generic-fw_dynamic.bin
Signature · 1 line
2.26.2
Re: [PATCH-for-5.2 1/2] gitlab-ci: Avoid running the OpenSBI job when not necessary
$SUBJECT seems to contradict the comment message and code.
Shouldn't it say "always run the OpenSBI job" since you're removing all the conditional logic ?
Quoted 62 lines · Philippe Mathieu-Daudé
The OpenSBI jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [*]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
As this job doesn't take much (less than 1min), run it by default.
[*] https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
Fixes 922febe2af ("Move edk2 and opensbi YAML files to .gitlab-ci.d folder")Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/opensbi.yml | 13 ------------- 1 file changed, 13 deletions(-).gitlab-ci.d/opensbi.yml +0
-13@@ -1,10 +1,5 @@ docker-opensbi: stage: containers - rules: # Only run this job when the Dockerfile is modified - - changes: - - .gitlab-ci.d/opensbi.yml - - .gitlab-ci.d/opensbi/Dockerfile - when: always image: docker:19.03.1 services: - docker:19.03.1-dind@@ -25,14 +20,6 @@ docker-opensbi: build-opensbi: stage: build - rules: # Only run this job when ... - - changes: # ... roms/opensbi/ is modified (submodule updated) - - roms/opensbi/* - when: always - - if: '$CI_COMMIT_REF_NAME =~ /^opensbi/' # or the branch/tag starts with 'opensbi' - when: always - - if: '$CI_COMMIT_MESSAGE =~ /opensbi/i' # or last commit description contains 'OpenSBI' - when: always artifacts: paths: # 'artifacts.zip' will contains the following files: - pc-bios/opensbi-riscv32-generic-fw_dynamic.binSignature · 1 line
2.26.2
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH-for-5.2 1/2] gitlab-ci: Avoid running the OpenSBI job when not necessary
Quoted 4 lines · Daniel P. Berrangé
$SUBJECT seems to contradict the comment message and code.
Shouldn't it say "always run the OpenSBI job" since you're removing all the conditional logic ?
Sorry, bad copy/paste from previous patch.
Quoted 68 lines · Philippe Mathieu-Daudé
Quoted 62 lines
The OpenSBI jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [*]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
As this job doesn't take much (less than 1min), run it by default.
[*] https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
Fixes 922febe2af ("Move edk2 and opensbi YAML files to .gitlab-ci.d folder")Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/opensbi.yml | 13 ------------- 1 file changed, 13 deletions(-).gitlab-ci.d/opensbi.yml +0
-13@@ -1,10 +1,5 @@ docker-opensbi: stage: containers - rules: # Only run this job when the Dockerfile is modified - - changes: - - .gitlab-ci.d/opensbi.yml - - .gitlab-ci.d/opensbi/Dockerfile - when: always image: docker:19.03.1 services: - docker:19.03.1-dind@@ -25,14 +20,6 @@ docker-opensbi: build-opensbi: stage: build - rules: # Only run this job when ... - - changes: # ... roms/opensbi/ is modified (submodule updated) - - roms/opensbi/* - when: always - - if: '$CI_COMMIT_REF_NAME =~ /^opensbi/' # or the branch/tag starts with 'opensbi' - when: always - - if: '$CI_COMMIT_MESSAGE =~ /opensbi/i' # or last commit description contains 'OpenSBI' - when: always artifacts: paths: # 'artifacts.zip' will contains the following files: - pc-bios/opensbi-riscv32-generic-fw_dynamic.binSignature · 1 line
2.26.2Regards,
Daniel
PATCH 2/2 gitlab-ci: Avoid running the EDK2 job when not necessary
In reply to Philippe Mathieu-DaudéThe EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
In commit 922febe2af we moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.
We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This jobs takes ~40min, and needlessly burns the 2000 minutes available to GitLab free users. Follow the recommendations in
[3] and disable this job by default (except if we push a tag or the branch contains 'edk2'). Note we do not remove the job from the pipeline, it can still be triggered manually from the WebUI.
[1] https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
[2] https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/
[3] https://about.gitlab.com/pricing/faq-consumption-cicd/
--- .gitlab-ci.d/edk2.yml | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
.gitlab-ci.d/edk2.yml +4 -7
@@ -1,10 +1,5 @@ docker-edk2: stage: containers - rules: # Only run this job when the Dockerfile is modified - - changes: - - .gitlab-ci.d/edk2.yml - - .gitlab-ci.d/edk2/Dockerfile - when: always image: docker:19.03.1 services: - docker:19.03.1-dind
@@ -26,13 +21,15 @@ docker-edk2: build-edk2: stage: build rules: # Only run this job when ... - - changes: # ... roms/edk2/ is modified (submodule updated) - - roms/edk2/* + - if: $CI_COMMIT_TAG # we pushed a tag when: always - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2' when: always - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2' when: always + # ... else allow manual run on the WebUI + - when: manual + allow_failure: true artifacts: paths: # 'artifacts.zip' will contains the following files: - pc-bios/edk2*bz2
Signature · 1 line
2.26.2
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
Something is fishy here and clearly not working, so clearly changes are needed, but the commit message is not explaining it for me.
Quoted 53 lines
This jobs takes ~40min, and needlessly burns the 2000 minutes available to GitLab free users. Follow the recommendations in
[3] and disable this job by default (except if we push a tag or the branch contains 'edk2'). Note we do not remove the job from the pipeline, it can still be triggered manually from the WebUI.[1] https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
[2] https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/
[3] https://about.gitlab.com/pricing/faq-consumption-cicd/Fixes 922febe2af ("Move edk2 and opensbi YAML files to .gitlab-ci.d folder")Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/edk2.yml | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-).gitlab-ci.d/edk2.yml +4
-7@@ -1,10 +1,5 @@ docker-edk2: stage: containers - rules: # Only run this job when the Dockerfile is modified - - changes: - - .gitlab-ci.d/edk2.yml - - .gitlab-ci.d/edk2/Dockerfile - when: always image: docker:19.03.1 services: - docker:19.03.1-dind@@ -26,13 +21,15 @@ docker-edk2: build-edk2: stage: build rules: # Only run this job when ... - - changes: # ... roms/edk2/ is modified (submodule updated) - - roms/edk2/* + - if: $CI_COMMIT_TAG # we pushed a tag when: always - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2' when: always - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2' when: always + # ... else allow manual run on the WebUI + - when: manual + allow_failure: true artifacts: paths: # 'artifacts.zip' will contains the following files: - pc-bios/edk2*bz2Signature · 1 line
2.26.2
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 27 lines · Daniel P. Berrangé
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
Quoted 6 lines
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
What about this one?
https://gitlab.com/berrange/qemu/-/jobs/827459510
Quoted 11 lines
Something is fishy here and clearly not working, so clearly changes are needed, but the commit message is not explaining it for me.
Quoted 6 lines
This jobs takes ~40min, and needlessly burns the 2000 minutes available to GitLab free users. Follow the recommendations in
[3] and disable this job by default (except if we push a tag or the branch contains 'edk2'). Note we do not remove the job from the pipeline, it can still be triggered manually from the WebUI.
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 32 lines · Philippe Mathieu-Daudé
Quoted 31 lines · Daniel P. Berrangé
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Quoted 6 lines
Quoted 2 lines
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
What about this one?
Ping?
Quoted 12 lines
Quoted 12 lines
Something is fishy here and clearly not working, so clearly changes are needed, but the commit message is not explaining it for me.
Quoted 6 lines
This jobs takes ~40min, and needlessly burns the 2000 minutes available to GitLab free users. Follow the recommendations in
[3] and disable this job by default (except if we push a tag or the branch contains 'edk2'). Note we do not remove the job from the pipeline, it can still be triggered manually from the WebUI.
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 33 lines · Philippe Mathieu-Daudé
Quoted 32 lines · Philippe Mathieu-Daudé
Quoted 31 lines · Daniel P. Berrangé
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
What about this one?
Ping?
Daniel's comment sounded like there are some changes required here, at least to improve the commit message? So I was expecting a v2 here. Please clarify Daniel's concerns, then I can either pick up this version here or the v2 (in case you send one).
Thomas
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 39 lines · Thomas Huth
Quoted 38 lines · Philippe Mathieu-Daudé
Quoted 37 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Daniel P. Berrangé
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
Quoted 6 lines
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
What about this one?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Ping?
Daniel's comment sounded like there are some changes required here, at least to improve the commit message? So I was expecting a v2 here. Please clarify Daniel's concerns, then I can either pick up this version here or the v2 (in case you send one).
Sorry, this "ping" was for Daniel. I'm not sure how to continue, so shared an example in his namespace of "incorrect behavior".
Meanwhile I'm tempted to put those jobs in Manual mode (2 months now), until we settle this with Daniel.
Thanks,
Phil.
Re: [PATCH-for-5.2 2/2] gitlab-ci: Avoid running the EDK2 job when not necessary
Quoted 44 lines · Philippe Mathieu-Daudé
Quoted 43 lines · Thomas Huth
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 32 lines · Philippe Mathieu-Daudé
Quoted 31 lines · Daniel P. Berrangé
Quoted 11 lines · Philippe Mathieu-Daudé
The EDK2 jobs use the 'changes' keyword, which "makes it possible to define if a job should be created based on files modified by a Git push event." (see [1]). This keyword comes with a warning:
Caution:
In pipelines with sources other than the three above changes can’t determine if a given file is new or old and always returns true."
Quoted 6 lines
In commit
922febe2afwe moved the YAML config file from the repository root directory to the .gitlab-ci.d/ directory.We didn't respect the previous warning and disabled the 'changes' filter rule, as the files are now in a (directory) three above the YAML config file.
This description is a bit wierd. I don't see how the location in the directory tree has any relevance here.
IIUC the caution docs quoted above are referring to what triggered the pipeline. They're saying that if the trigger was not a "branch", "merge request", or "external pull request", then the "changes" rule always evaluates true.
The "branch" source us a bit wierd though, as I'm not seeing how gitlab figures out which commits are "new" to the pipeline and thus whether the files were modified or not.
Quoted 6 lines
Quoted 2 lines
Strangely qemu-project/qemu CI for master seems to be behaving correctly and skipping the jobs.
What about this one?
Ping?
Quoted 7 lines
Quoted 4 lines
Daniel's comment sounded like there are some changes required here, at least to improve the commit message? So I was expecting a v2 here. Please clarify Daniel's concerns, then I can either pick up this version here or the v2 (in case you send one).
Sorry, this "ping" was for Daniel. I'm not sure how to continue, so shared an example in his namespace of "incorrect behavior".
Looking at pipelines than ran for me last week, I'm not seeing ED2 being run, likewise for recent git master jobs on the main repo.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|