Skip to contentSkip to conversation

gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

RFC PATCH Superseded Thomas Huth 2 participants 5 messages May 3, 2022 – May 16, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 gitlab-ci: Switch the 'check-patch' & 'check-dco' jobs to use python-container

The 'check-patch' and 'check-dco' jobs only need Python and git for checking the patches, so it's not really necessary to use a container here that has all the other build dependencies installed. By installing "git" in the python container, we can use this light-weight container for these jobs instead.

---
Mark as RFC since I'm not sure whether we want to have "git" in the python container or not?

 .gitlab-ci.d/static_checks.yml         | 8 ++++----
 tests/docker/dockerfiles/python.docker | 1 +
 2 files changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/static_checks.yml +4 -4
index 5e955540d3..0e080bd0a0 100644
@@ -1,8 +1,8 @@
 check-patch:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script:
     - .gitlab-ci.d/check-patch.py
   variables:
@@ -15,9 +15,9 @@ check-patch:
 
 check-dco:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script: .gitlab-ci.d/check-dco.py
   variables:
     GIT_DEPTH: 1000
tests/docker/dockerfiles/python.docker +1 -0
index 56d88417df..b2fb3a306d 100644
@@ -6,6 +6,7 @@ MAINTAINER John Snow <[email protected]>
 # Please keep this list sorted alphabetically
 ENV PACKAGES \
     gcc \
+    git \
     make \
     pipenv \
     python3 \
Signature · 1 line
2.27.0
#1 lore raw
Quoted 5 lines · Thomas Huth

The 'check-patch' and 'check-dco' jobs only need Python and git for checking the patches, so it's not really necessary to use a container here that has all the other build dependencies installed. By installing "git" in the python container, we can use this light-weight container for these jobs instead.

Our python container is far from light-weight....

$ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python

real 1m52.717s
user 1m32.327s
sys 0m19.453s

vs

$ time podman pull python:3.8-alpine

real 0m4.509s
user 0m3.780s
sys 0m1.052s

It is quicker to use the alpine python container and then just install 'git' on every job, than it is to use the pre-built qemu python container

Quoted 27 lines

---
Mark as RFC since I'm not sure whether we want to have "git" in the python container or not?

 .gitlab-ci.d/static_checks.yml         | 8 ++++----
 tests/docker/dockerfiles/python.docker | 1 +
 2 files changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/static_checks.yml +2 -2
index 5e955540d3..0e080bd0a0 100644
@@ -1,8 +1,8 @@
 check-patch:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script:
     - .gitlab-ci.d/check-patch.py
   variables:
@@ -15,9 +15,9 @@ check-patch:
 

.gitlab-ci.d/static_checks.yml check-patch: · line 23

Quoted 10 lines
.gitlab-ci.d/static_checks.yml +2 -2
 check-dco:
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
   needs:
-    job: amd64-centos8-container
+    job: python-container
   script: .gitlab-ci.d/check-dco.py
   variables:
     GIT_DEPTH: 1000

IOW this is sufficient:

image: python:3.8-alpine
needs: []
before_script:
- apk update
- apk add git
script: ./gitlab-ci.d/check-dco.py

I expect the same would work for check-patch.py container

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
Quoted 76 lines · Daniel P. Berrangé
Quoted 5 lines · Thomas Huth

The 'check-patch' and 'check-dco' jobs only need Python and git for checking the patches, so it's not really necessary to use a container here that has all the other build dependencies installed. By installing "git" in the python container, we can use this light-weight container for these jobs instead.

Our python container is far from light-weight....

$ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python

real 1m52.717s
user 1m32.327s
sys 0m19.453s

vs

$ time podman pull python:3.8-alpine

real 0m4.509s
user 0m3.780s
sys 0m1.052s

It is quicker to use the alpine python container and then just install 'git' on every job, than it is to use the pre-built qemu python container

Quoted 27 lines

---
Mark as RFC since I'm not sure whether we want to have "git" in the python container or not?

  .gitlab-ci.d/static_checks.yml         | 8 ++++----
  tests/docker/dockerfiles/python.docker | 1 +
  2 files changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/static_checks.yml +2 -2
index 5e955540d3..0e080bd0a0 100644
@@ -1,8 +1,8 @@
  check-patch:
    stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
    needs:
-    job: amd64-centos8-container
+    job: python-container
    script:
      - .gitlab-ci.d/check-patch.py
    variables:
@@ -15,9 +15,9 @@ check-patch:
 

.gitlab-ci.d/static_checks.yml check-patch: · line 23

Quoted 10 lines
.gitlab-ci.d/static_checks.yml +2 -2
  check-dco:
    stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
    needs:
-    job: amd64-centos8-container
+    job: python-container
    script: .gitlab-ci.d/check-dco.py
    variables:
      GIT_DEPTH: 1000

IOW this is sufficient:

image: python:3.8-alpine
needs: []
before_script:
- apk update
- apk add git
script: ./gitlab-ci.d/check-dco.py

Quoted 1 line

I expect the same would work for check-patch.py container

... or would it make sense to switch tests/docker/dockerfiles/python.docker to use alpine instead of fedora?

Thomas

#3 lore raw
Quoted 76 lines · Thomas Huth
Quoted 75 lines · Daniel P. Berrangé
Quoted 5 lines · Thomas Huth

The 'check-patch' and 'check-dco' jobs only need Python and git for checking the patches, so it's not really necessary to use a container here that has all the other build dependencies installed. By installing "git" in the python container, we can use this light-weight container for these jobs instead.

Our python container is far from light-weight....

$ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python

real 1m52.717s
user 1m32.327s
sys 0m19.453s

vs

$ time podman pull python:3.8-alpine

real 0m4.509s
user 0m3.780s
sys 0m1.052s

It is quicker to use the alpine python container and then just install 'git' on every job, than it is to use the pre-built qemu python container

Quoted 26 lines

---
Mark as RFC since I'm not sure whether we want to have "git" in the python container or not?

  .gitlab-ci.d/static_checks.yml         | 8 ++++----
  tests/docker/dockerfiles/python.docker | 1 +
  2 files changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/static_checks.yml +2 -2
index 5e955540d3..0e080bd0a0 100644
@@ -1,8 +1,8 @@
  check-patch:
    stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
    needs:
-    job: amd64-centos8-container
+    job: python-container
    script:
      - .gitlab-ci.d/check-patch.py
    variables:

.gitlab-ci.d/static_checks.yml check-patch: · line 22

Quoted 10 lines
.gitlab-ci.d/static_checks.yml +2 -2
@@ -15,9 +15,9 @@ check-patch:
  check-dco:
    stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
    needs:
-    job: amd64-centos8-container
+    job: python-container
    script: .gitlab-ci.d/check-dco.py
    variables:
      GIT_DEPTH: 1000

IOW this is sufficient:

image: python:3.8-alpine
needs: []
before_script:
- apk update
- apk add git
script: ./gitlab-ci.d/check-dco.py

Quoted 4 lines
Quoted 1 line

I expect the same would work for check-patch.py container

... or would it make sense to switch tests/docker/dockerfiles/python.docker to use alpine instead of fedora?

That could work too

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 77 lines · Daniel P. Berrangé
Quoted 76 lines · Thomas Huth
Quoted 75 lines · Daniel P. Berrangé
Quoted 5 lines · Thomas Huth

The 'check-patch' and 'check-dco' jobs only need Python and git for checking the patches, so it's not really necessary to use a container here that has all the other build dependencies installed. By installing "git" in the python container, we can use this light-weight container for these jobs instead.

Our python container is far from light-weight....

$ time podman pull registry.gitlab.com/qemu-project/qemu/qemu/python

real 1m52.717s
user 1m32.327s
sys 0m19.453s

vs

$ time podman pull python:3.8-alpine

real 0m4.509s
user 0m3.780s
sys 0m1.052s

It is quicker to use the alpine python container and then just install 'git' on every job, than it is to use the pre-built qemu python container

Quoted 26 lines

---
Mark as RFC since I'm not sure whether we want to have "git" in the python container or not?

   .gitlab-ci.d/static_checks.yml         | 8 ++++----
   tests/docker/dockerfiles/python.docker | 1 +
   2 files changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/static_checks.yml +2 -2
index 5e955540d3..0e080bd0a0 100644
@@ -1,8 +1,8 @@
   check-patch:
     stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
     needs:
-    job: amd64-centos8-container
+    job: python-container
     script:
       - .gitlab-ci.d/check-patch.py
     variables:

.gitlab-ci.d/static_checks.yml check-patch: · line 22

Quoted 10 lines
.gitlab-ci.d/static_checks.yml +2 -2
@@ -15,9 +15,9 @@ check-patch:
   check-dco:
     stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:latest
     needs:
-    job: amd64-centos8-container
+    job: python-container
     script: .gitlab-ci.d/check-dco.py
     variables:
       GIT_DEPTH: 1000

IOW this is sufficient:

image: python:3.8-alpine
needs: []
before_script:
- apk update
- apk add git
script: ./gitlab-ci.d/check-dco.py

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I expect the same would work for check-patch.py container

... or would it make sense to switch tests/docker/dockerfiles/python.docker to use alpine instead of fedora?

That could work too

I just tried it, and it did not work out as expected: The check-python-pipenv job needs a more sophisticated environment than the alpine container can provide easily (e.g. it still needs python-3.6), so I updated my patch with your idea instead and used the python:alpine container as a base there now.

Thomas