support/misc/gitlab-ci.yml.in: include Branches workflow
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a when clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].
Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
---
I'd like to get this one in master, because the e-mails are annoying me :-)
--- support/misc/gitlab-ci.yml.in | 3 +++ 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' + image: buildroot/base:20200814.2228 .check_base:
Signature · 1 line
2.26.2
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
Arnout, All,
+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.
On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
Quoted 11 lines
Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a
whenclause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.
Quoted 8 lines
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds---
I'd like to get this one in master, because the e-mails are annoying me :-)
I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.
Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.
Also, this will allow us to not care about the evolutions of that micro- language.
To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.
Regards,
Yann E. MORIN.
Quoted 25 lines
--- support/misc/gitlab-ci.yml.in | 3 +++ 1 file changed, 3 insertions(+)support/misc/gitlab-ci.yml.in +3
-0@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' + image: buildroot/base:20200814.2228 .check_base:Signature · 6 lines
2.26.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
Quoted 21 lines · Yann E. MORIN
Arnout, All,
+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.
On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
Quoted 15 lines
Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a
whenclause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
Quoted 8 lines
Quoted 4 lines
---
I'd like to get this one in master, because the e-mails are annoying me :-)I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.
Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).
Also, I would like to move the "constant" bits back out to the top-level
gitlab-ci.yml, so the first page already shows the quick test results.
Quoted 7 lines
Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.
Also, this will allow us to not care about the evolutions of that micro- language.
Quoted 7 lines
To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.
I think that's a great idea. However, that sounds like a bigger change which
probably shouldn't go into master. So I'd prefer to still merge my patch as well.
Regards,
Arnout
Quoted 30 lines
Regards,
Yann E. MORIN.Quoted 25 lines
--- support/misc/gitlab-ci.yml.in | 3 +++ 1 file changed, 3 insertions(+)support/misc/gitlab-ci.yml.in +3
-0@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' + image: buildroot/base:20200814.2228 .check_base:Signature · 6 lines
2.26.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
Arnout, All,
On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
Quoted 4 lines
Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).
Sorry, but I don't follow: if the jobs are only generated when we want them, then they no longer have conditions in the generated gitlab-ci.yml to begin with; that's the whole point of moving the conditional handling into the script.
Quoted 4 lines · Yann E. MORIN
Quoted 3 lines
I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.
[--SNIP--]
Quoted 2 lines
Also, I would like to move the "constant" bits back out to the top-level
gitlab-ci.yml, so the first page already shows the quick test results.
Sorry, but again they are not "constant". For example, check-flake8, check-package and check-DEVELOPERS are not run when one pushes a *_defconfig or *_test branch (or however they are named) to trigger the build of jsut that defconfig or just that runtime test.
Quoted 8 lines
Quoted 6 lines
Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.
Also, this will allow us to not care about the evolutions of that micro- language.
I think that's a great idea. However, that sounds like a bigger change which
probably shouldn't go into master. So I'd prefer to still merge my patch as well.
See an actual comment on that patch, below...
Quoted 10 lines
Quoted 10 lines
support/misc/gitlab-ci.yml.in line 5
Quoted 10 lines
support/misc/gitlab-ci.yml.in +2
-0@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
Where does that file come from? How are we guaranteed that its content does not change and break our pipelines?
If that template is interesting, then we should carry it rather than depend on an external, un-managed entity...
Regards,
Yann E. MORIN.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
support/misc/gitlab-ci.yml.in +0
-0image: buildroot/base:20200814.2228 .check_base:Signature · 6 lines
2.26.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
Quoted 8 lines · Yann E. MORIN
Arnout, All,
On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
Quoted 4 lines
Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).
Quoted 4 lines
Sorry, but I don't follow: if the jobs are only generated when we want them, then they no longer have conditions in the generated gitlab-ci.yml to begin with; that's the whole point of moving the conditional handling into the script.
Ah, OK, I thought it was just about generating the defconfig and tests parts.
My bad.
Quoted 1 line
Quoted 1 line · Yann E. MORIN
[snip]
Quoted 11 lines
See an actual comment on that patch, below...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
support/misc/gitlab-ci.yml.in +0
-0@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
support/misc/gitlab-ci.yml.in line 5
Quoted 2 lines
support/misc/gitlab-ci.yml.in +2
-0+include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'Where does that file come from? How are we guaranteed that its content does not change and break our pipelines?
If that template is interesting, then we should carry it rather than depend on an external, un-managed entity...
It is part of the gitlab installation. That means that indeed it is not under
our control, but neither is the rest of gitlab (e.g., the part that generates those warnings :-). In other words: it's about as reliable as the rules clauses we have now.
Note that the workflow file is indeed very simple, and we could simply
instantiate it here.
Again, though, in order to expedite the 2020.08 release, I propose to apply
this simple patch to master, and soon after merge your series that makes it script-based - possibly backporting it to the stable branches, though I don't really think that is needed.
Regards,
Arnout
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
In reply to Arnout VandecappelleArnout, Jugurtha, All,
On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
Quoted 3 lines · Yann E. MORIN
Quoted 2 lines
+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.
[--SNIP--]
Quoted 5 lines
Quoted 5 lines
To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.
And of course, as usual, I did not have time to properly finish that branch...
Still, it is very close to be ready, with all the test cases now working as expected:
Random branch, only basic tests
https://gitlab.com/ymorin/buildroot/-/commits/yem/gitlab-ci-cond https://gitlab.com/ymorin/buildroot/-/pipelines/184068345
All defconfigs:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-defconfigs https://gitlab.com/ymorin/buildroot/-/pipelines/184069188
One defconfig:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-warp7_defconfig https://gitlab.com/ymorin/buildroot/-/pipelines/184068523
All runtime tests:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-runtime-tests https://gitlab.com/ymorin/buildroot/-/pipelines/184070380
One runtime test:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-tests.package.test_openssh https://gitlab.com/ymorin/buildroot/-/pipelines/184068633
I still have to test a maintenance branch (by faking one with a branch named 'fooo.no.x' to match the pattern), but I'm afraid I've reached my allowed number of jobs for now...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow
In reply to Arnout Vandecappelle (Essensium/Mind)Arnout, All,
On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
Quoted 11 lines
Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a
whenclause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.
Quoted 5 lines
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
We've no merged a better script that contains all the logic about generating the child pipeline, so I believe this patch of yours is no longer needed.
Regards,
Yann E. MORIN.
Quoted 28 lines
---
I'd like to get this one in master, because the e-mails are annoying me :-)--- support/misc/gitlab-ci.yml.in | 3 +++ 1 file changed, 3 insertions(+)support/misc/gitlab-ci.yml.in +3
-0@@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' + image: buildroot/base:20200814.2228 .check_base:Signature · 6 lines
2.26.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'