Skip to contentSkip to conversation

support/misc/gitlab-ci.yml.in: include Branches workflow

PATCH New Arnout Vandecappelle (Essensium/Mind) 2 participants 7 messages Aug 30, 2020 – Oct 6, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a when clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].

Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds

---
I'd like to get this one in master, because the e-mails are annoying me :-)

---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
+
 image: buildroot/base:20200814.2228
 
 .check_base:
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

Arnout, All,

+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.

On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:

Quoted 11 lines

Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a when clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].

Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.

Quoted 8 lines

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds

---
I'd like to get this one in master, because the e-mails are annoying me :-)

I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.

Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.

Also, this will allow us to not care about the evolutions of that micro- language.

To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond

I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.

Regards,
Yann E. MORIN.

Quoted 25 lines
---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
+
 image: buildroot/base:20200814.2228
 
 .check_base:
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

Quoted 21 lines · Yann E. MORIN

Arnout, All,

+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.

On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:

Quoted 15 lines

Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a when clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].

Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds

Quoted 8 lines
Quoted 4 lines

---
I'd like to get this one in master, because the e-mails are annoying me :-)

I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.

Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).

Also, I would like to move the "constant" bits back out to the top-level
gitlab-ci.yml, so the first page already shows the quick test results.

Quoted 7 lines

Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.

Also, this will allow us to not care about the evolutions of that micro- language.

Quoted 7 lines

To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond

I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.

I think that's a great idea. However, that sounds like a bigger change which
probably shouldn't go into master. So I'd prefer to still merge my patch as well.

Regards,
Arnout

Quoted 30 lines

Regards,
Yann E. MORIN.

Quoted 25 lines
---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
+
 image: buildroot/base:20200814.2228
 
 .check_base:
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
#3 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

Arnout, All,

On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:

Quoted 4 lines

Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).

Sorry, but I don't follow: if the jobs are only generated when we want them, then they no longer have conditions in the generated gitlab-ci.yml to begin with; that's the whole point of moving the conditional handling into the script.

Quoted 4 lines · Yann E. MORIN
Quoted 3 lines

I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description.

[--SNIP--]

Quoted 2 lines

Also, I would like to move the "constant" bits back out to the top-level
gitlab-ci.yml, so the first page already shows the quick test results.

Sorry, but again they are not "constant". For example, check-flake8, check-package and check-DEVELOPERS are not run when one pushes a *_defconfig or *_test branch (or however they are named) to trigger the build of jsut that defconfig or just that runtime test.

Quoted 8 lines
Quoted 6 lines

Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml.

Also, this will allow us to not care about the evolutions of that micro- language.

I think that's a great idea. However, that sounds like a bigger change which
probably shouldn't go into master. So I'd prefer to still merge my patch as well.

See an actual comment on that patch, below...

Quoted 10 lines
Quoted 10 lines

support/misc/gitlab-ci.yml.in line 5

Quoted 10 lines
support/misc/gitlab-ci.yml.in +2 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'

Where does that file come from? How are we guaranteed that its content does not change and break our pipelines?

If that template is interesting, then we should carry it rather than depend on an external, un-managed entity...

Regards,
Yann E. MORIN.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
support/misc/gitlab-ci.yml.in +0 -0
 image: buildroot/base:20200814.2228
 
 .check_base:
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

Quoted 8 lines · Yann E. MORIN

Arnout, All,

On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:

Quoted 4 lines

Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches).

Quoted 4 lines

Sorry, but I don't follow: if the jobs are only generated when we want them, then they no longer have conditions in the generated gitlab-ci.yml to begin with; that's the whole point of moving the conditional handling into the script.

Ah, OK, I thought it was just about generating the defconfig and tests parts.
My bad.

Quoted 1 line
Quoted 1 line · Yann E. MORIN

[snip]

Quoted 11 lines

See an actual comment on that patch, below...

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
support/misc/gitlab-ci.yml.in +0 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

support/misc/gitlab-ci.yml.in line 5

Quoted 2 lines
support/misc/gitlab-ci.yml.in +2 -0
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'

Where does that file come from? How are we guaranteed that its content does not change and break our pipelines?

If that template is interesting, then we should carry it rather than depend on an external, un-managed entity...

It is part of the gitlab installation. That means that indeed it is not under
our control, but neither is the rest of gitlab (e.g., the part that generates those warnings :-). In other words: it's about as reliable as the rules clauses we have now.

Note that the workflow file is indeed very simple, and we could simply
instantiate it here.

Again, though, in order to expedite the 2020.08 release, I propose to apply
this simple patch to master, and soon after merge your series that makes it script-based - possibly backporting it to the stable branches, though I don't really think that is needed.

Regards,
Arnout

#5 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

In reply to Arnout Vandecappelle

Arnout, Jugurtha, All,

On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:

Quoted 3 lines · Yann E. MORIN
Quoted 2 lines

+Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before.

[--SNIP--]

Quoted 5 lines
Quoted 5 lines

To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think):
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight.

And of course, as usual, I did not have time to properly finish that branch...

Still, it is very close to be ready, with all the test cases now working as expected:

Random branch, only basic tests

    https://gitlab.com/ymorin/buildroot/-/commits/yem/gitlab-ci-cond
    https://gitlab.com/ymorin/buildroot/-/pipelines/184068345

All defconfigs:

    https://gitlab.com/ymorin/buildroot/-/commits/FOO-defconfigs
    https://gitlab.com/ymorin/buildroot/-/pipelines/184069188

One defconfig:

    https://gitlab.com/ymorin/buildroot/-/commits/FOO-warp7_defconfig
    https://gitlab.com/ymorin/buildroot/-/pipelines/184068523

All runtime tests:

    https://gitlab.com/ymorin/buildroot/-/commits/FOO-runtime-tests
    https://gitlab.com/ymorin/buildroot/-/pipelines/184070380

One runtime test:

    https://gitlab.com/ymorin/buildroot/-/commits/FOO-tests.package.test_openssh
    https://gitlab.com/ymorin/buildroot/-/pipelines/184068633

I still have to test a maintenance branch (by faking one with a branch named 'fooo.no.x' to match the pattern), but I'm afraid I've reached my allowed number of jobs for now...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH 1/1 support/misc/gitlab-ci.yml.in: include Branches workflow

In reply to Arnout Vandecappelle (Essensium/Mind)

Arnout, All,

On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:

Quoted 11 lines

Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a when clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2].

Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this.

Quoted 5 lines

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds

We've no merged a better script that contains all the logic about generating the child pipeline, so I believe this patch of yours is no longer needed.

Regards,
Yann E. MORIN.

Quoted 28 lines

---
I'd like to get this one in master, because the e-mails are annoying me :-)

---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
index dddebf09e9..8a031898ef 100644
@@ -1,6 +1,9 @@
 # Configuration for Gitlab-CI.
 # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
 
+include:
+  - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
+
 image: buildroot/base:20200814.2228
 
 .check_base:
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'