Skip to contentSkip to conversation

gitlab-ci: fix pipelines with the newer docker image (branch yem/pipelines)

PATCH In review Yann E. MORIN 2 participants 7 messages Feb 7, 2023

lore
Share & tools
lore
#0 lore raw

cover gitlab-ci: fix pipelines with the newer docker image (branch yem/pipelines)

Hello All!

TL;DR: the newer docker image we use in our pipelines uses a newer git version that is more strict with the ownership of the files in the git repository checkout, which makes our check jobs fail.

Long story: see the individual commits. ;-)

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (2):
gitlab-ci: don't use before_script in job templates
gitlab-ci: fix pipelines

 support/misc/gitlab-ci.yml.in | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH 1/2 gitlab-ci: don't use before_script in job templates

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore the situation. In either case, we'll need to run a scriptlet before all our jobs.

Gitlab-ci allows to provide a global before_script, that is inherited by all jobs. However, some of our jobs already declare a before_script, and that would shadow the global before_Scrikpt.

There is no technical reason to do our before_script separately from the actual script, so we move the code from the before_scripts to the corresponding scripts.

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
support/misc/gitlab-ci.yml.in +4 -8
index 0ccf36665e..9c1faf0d5f 100644
@@ -23,9 +23,8 @@
         - utils/check-symbols
 
 .defconfig_check:
-    before_script:
+    script:
         - DEFCONFIG_NAME=$(echo ${CI_JOB_NAME} | sed -e 's,_check$,,g')
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - support/scripts/check-dotconfig.py .config configs/${DEFCONFIG_NAME}
@@ -44,10 +43,9 @@
       }
 
 .defconfig_base:
-    before_script:
+    script:
         - DEFCONFIG_NAME=${CI_JOB_NAME}
         - OUTPUT_DIR=output
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
@@ -72,13 +70,12 @@
             - runtime-test.log
 
 .runtime_test_base:
-    before_script:
-        - TEST_CASE_NAME=${CI_JOB_NAME}
     # Keep build directories so the rootfs can be an artifact of the job. The
     # runner will clean up those files for us.
     # Multiply every emulator timeout by 10 to avoid sporadic failures in
     # elastic runners.
     script:
+        - TEST_CASE_NAME=${CI_JOB_NAME}
         - echo "Starting runtime test ${TEST_CASE_NAME}"
         - ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${TEST_CASE_NAME}
     artifacts:
@@ -91,9 +88,8 @@
 
 .test_pkg:
     stage: build
-    before_script:
+    script:
         - OUTPUT_DIR=${CI_JOB_NAME}
-    script:
         - echo "Configure Buildroot for ${OUTPUT_DIR}"
         - make O=${OUTPUT_DIR} syncconfig
         - make O=${OUTPUT_DIR} savedefconfig
Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by on v1 1/2
#2 lore raw

Re: [Buildroot] [PATCH 1/2] gitlab-ci: don't use before_script in job templates

Hello Yann,

Le 07/02/2023 à 22:33, Yann E. MORIN a écrit :

Quoted 8 lines

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

Quoted 6 lines

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

is flake8 has been updated too? flake8 may trigger new check-package warning now.

Quoted 8 lines

To fix this, we either need to fix the ownership or tell git to ignore the situation. In either case, we'll need to run a scriptlet before all our jobs.

Gitlab-ci allows to provide a global before_script, that is inherited by all jobs. However, some of our jobs already declare a before_script, and that would shadow the global before_Scrikpt.

before_Scrikpt/before_scripts

Quoted 5 lines

There is no technical reason to do our before_script separately from the actual script, so we move the code from the before_scripts to the corresponding scripts.

Best regards,
Romain

Quoted 51 lines
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
support/misc/gitlab-ci.yml.in +3 -6
index 0ccf36665e..9c1faf0d5f 100644
@@ -23,9 +23,8 @@
         - utils/check-symbols
 
 .defconfig_check:
-    before_script:
+    script:
         - DEFCONFIG_NAME=$(echo ${CI_JOB_NAME} | sed -e 's,_check$,,g')
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - support/scripts/check-dotconfig.py .config configs/${DEFCONFIG_NAME}
@@ -44,10 +43,9 @@
       }
 
 .defconfig_base:
-    before_script:
+    script:
         - DEFCONFIG_NAME=${CI_JOB_NAME}
         - OUTPUT_DIR=output
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
@@ -72,13 +70,12 @@
             - runtime-test.log
 
 .runtime_test_base:
-    before_script:
-        - TEST_CASE_NAME=${CI_JOB_NAME}
     # Keep build directories so the rootfs can be an artifact of the job. The
     # runner will clean up those files for us.
     # Multiply every emulator timeout by 10 to avoid sporadic failures in
     # elastic runners.
     script:
+        - TEST_CASE_NAME=${CI_JOB_NAME}
         - echo "Starting runtime test ${TEST_CASE_NAME}"
         - ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${TEST_CASE_NAME}
     artifacts:

support/misc/gitlab-ci.yml.in line 95

Quoted 10 lines
support/misc/gitlab-ci.yml.in +1 -2
@@ -91,9 +88,8 @@
 
 .test_pkg:
     stage: build
-    before_script:
+    script:
         - OUTPUT_DIR=${CI_JOB_NAME}
-    script:
         - echo "Configure Buildroot for ${OUTPUT_DIR}"
         - make O=${OUTPUT_DIR} syncconfig
         - make O=${OUTPUT_DIR} savedefconfig

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw

Re: [Buildroot] [PATCH 1/2] gitlab-ci: don't use before_script in job templates

Romain, All,

On 2023-02-07 22:47 +0100, Romain Naour spake thusly:

Quoted 9 lines

Le 07/02/2023 à 22:33, Yann E. MORIN a écrit :

Quoted 8 lines

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

Quoted 8 lines
Quoted 6 lines

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

is flake8 has been updated too? flake8 may trigger new check-package warning now.

Yes, there is a newer flake8 too, but there is no new error detected:

    https://gitlab.com/ymorin/buildroot/-/pipelines/770406457
Quoted 9 lines
Quoted 8 lines

To fix this, we either need to fix the ownership or tell git to ignore the situation. In either case, we'll need to run a scriptlet before all our jobs.

Gitlab-ci allows to provide a global before_script, that is inherited by all jobs. However, some of our jobs already declare a before_script, and that would shadow the global before_Scrikpt.

before_Scrikpt/before_scripts

Thanks Thomas said he'd fix that when applying. ;-]

Quoted 4 lines
Quoted 3 lines

There is no technical reason to do our before_script separately from the actual script, so we move the code from the before_scripts to the corresponding scripts.

Thanks!

Regards,
Yann E. MORIN.

Quoted 70 lines

Best regards,
Romain

Quoted 51 lines
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
support/misc/gitlab-ci.yml.in +3 -6
index 0ccf36665e..9c1faf0d5f 100644
@@ -23,9 +23,8 @@
         - utils/check-symbols
 
 .defconfig_check:
-    before_script:
+    script:
         - DEFCONFIG_NAME=$(echo ${CI_JOB_NAME} | sed -e 's,_check$,,g')
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - support/scripts/check-dotconfig.py .config configs/${DEFCONFIG_NAME}
@@ -44,10 +43,9 @@
       }
 
 .defconfig_base:
-    before_script:
+    script:
         - DEFCONFIG_NAME=${CI_JOB_NAME}
         - OUTPUT_DIR=output
-    script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
@@ -72,13 +70,12 @@
             - runtime-test.log
 
 .runtime_test_base:
-    before_script:
-        - TEST_CASE_NAME=${CI_JOB_NAME}
     # Keep build directories so the rootfs can be an artifact of the job. The
     # runner will clean up those files for us.
     # Multiply every emulator timeout by 10 to avoid sporadic failures in
     # elastic runners.
     script:
+        - TEST_CASE_NAME=${CI_JOB_NAME}
         - echo "Starting runtime test ${TEST_CASE_NAME}"
         - ./support/testing/run-tests -o test-output/ -d test-dl/ -k --timeout-multiplier 10 ${TEST_CASE_NAME}
     artifacts:

support/misc/gitlab-ci.yml.in line 95

Quoted 10 lines
support/misc/gitlab-ci.yml.in +1 -2
@@ -91,9 +88,8 @@
 
 .test_pkg:
     stage: build
-    before_script:
+    script:
         - OUTPUT_DIR=${CI_JOB_NAME}
-    script:
         - echo "Configure Buildroot for ${OUTPUT_DIR}"
         - make O=${OUTPUT_DIR} syncconfig
         - make O=${OUTPUT_DIR} savedefconfig

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

PATCH 2/2 gitlab-ci: fix pipelines

In reply to Yann E. MORIN

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore the situation.

It is most probably impossible to change the ownership of the files: we run as non-root,and the files belong to root (in the container). So we're stuck.

The alternative, is to do as git suggest, and tell it to ignore the situation. In a local setup, this woujld be very insecure, but in the pipelines, this is in a throw-away container, wehre a single user exists and is running, so we don't care much (if at all).

Add a global before_script that registers the git config to ignore ownership issues in the buildroot repository; see [0] for the definition of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and in [1], the value actually seen in CI_PROJECT_DIR is already prefixed with CI_BUILDS_DIR (the documentation is unclear about that point).

[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +3 -0
index 9c1faf0d5f..38aca31fb5 100644
@@ -1,3 +1,6 @@
+before_script:
+  - git config --global --add safe.directory ${CI_PROJECT_DIR}
+
 .check-check-package_base:
     script:
         - python3 -m pytest -v utils/checkpackagelib/
Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by on v1 2/2
#5 lore raw

Re: [Buildroot] [PATCH 2/2] gitlab-ci: fix pipelines

Hello Yann,

Is "fix pipelines" your WIP patch subject ? "gitlab-ci: ignore ownership of the git tree" seems better?

Le 07/02/2023 à 22:33, Yann E. MORIN a écrit :

Quoted 27 lines

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore the situation.

It is most probably impossible to change the ownership of the files: we run as non-root,and the files belong to root (in the container). So we're stuck.

The alternative, is to do as git suggest, and tell it to ignore the situation. In a local setup, this woujld be very insecure, but in the pipelines, this is in a throw-away container, wehre a single user exists and is running, so we don't care much (if at all).

Quoted 8 lines

Add a global before_script that registers the git config to ignore ownership issues in the buildroot repository; see [0] for the definition of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and in [1], the value actually seen in CI_PROJECT_DIR is already prefixed with CI_BUILDS_DIR (the documentation is unclear about that point).

[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

Quoted 14 lines
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +0 -0
index 9c1faf0d5f..38aca31fb5 100644

support/misc/gitlab-ci.yml.in line 2

Quoted 2 lines
support/misc/gitlab-ci.yml.in +2 -0
@@ -1,3 +1,6 @@
+before_script:
+  - git config --global --add safe.directory ${CI_PROJECT_DIR}

Note: I checked if we have the issue using the utils/docker-run directly, enabling safe.directory is not necessary.

Best regards,
Romain

support/misc/gitlab-ci.yml.in line 6

Quoted 4 lines
support/misc/gitlab-ci.yml.in +1 -0
+
 .check-check-package_base:
     script:
         - python3 -m pytest -v utils/checkpackagelib/

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#6 lore raw

Re: [Buildroot] [PATCH 2/2] gitlab-ci: fix pipelines

Romain, All,

On 2023-02-07 22:57 +0100, Romain Naour spake thusly:

Quoted 2 lines

Is "fix pipelines" your WIP patch subject ? "gitlab-ci: ignore ownership of the git tree" seems better?

No, I really intended to write "fix pipeline".

Indeed, the subject of a commit log should be a summary of the semantic change, not of how it is done.

Thanks for the review! :-)

Regards,
Yann E. MORIN.

Quoted 73 lines

Le 07/02/2023 à 22:33, Yann E. MORIN a écrit :

Quoted 27 lines

When gitlab prepares a job to run, it checks out the repository with a non-root user, and spawns a container that runs as root, with some UID mapping that makes the files be owned by root in the container. However, our pipelines run as a nont-root user.

Commit bde165f7ad (.gitlab-ci.yml: update Docker image to use) updated the docker image that is used to run in our pipelines.

That new image includes a git version that is stricter about the ownership of the git tree it is acting in: git aborts in error when the user running it does not own the repository.

We use git ls-tree quite a lot in our check-{flake8,package,symbols} rules, so they all fail (in various ways).

To fix this, we either need to fix the ownership or tell git to ignore the situation.

It is most probably impossible to change the ownership of the files: we run as non-root,and the files belong to root (in the container). So we're stuck.

The alternative, is to do as git suggest, and tell it to ignore the situation. In a local setup, this woujld be very insecure, but in the pipelines, this is in a throw-away container, wehre a single user exists and is running, so we don't care much (if at all).

Quoted 8 lines

Add a global before_script that registers the git config to ignore ownership issues in the buildroot repository; see [0] for the definition of the CI_PROJECT_DIR variable. Note: unlike what is said in there, and in [1], the value actually seen in CI_PROJECT_DIR is already prefixed with CI_BUILDS_DIR (the documentation is unclear about that point).

[0] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
[1] https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

Quoted 14 lines
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Romain Naour <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/misc/gitlab-ci.yml.in | 3 +++
 1 file changed, 3 insertions(+)
support/misc/gitlab-ci.yml.in +0 -0
index 9c1faf0d5f..38aca31fb5 100644

support/misc/gitlab-ci.yml.in line 2

Quoted 2 lines
support/misc/gitlab-ci.yml.in +2 -0
@@ -1,3 +1,6 @@
+before_script:
+  - git config --global --add safe.directory ${CI_PROJECT_DIR}

Note: I checked if we have the issue using the utils/docker-run directly, enabling safe.directory is not necessary.

Best regards,
Romain

support/misc/gitlab-ci.yml.in line 6

Quoted 4 lines
support/misc/gitlab-ci.yml.in +1 -0
+
 .check-check-package_base:
     script:
         - python3 -m pytest -v utils/checkpackagelib/

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot