Skip to contentSkip to conversation

gitlab: disable provenance attestations to work around CI bug

PATCH Applied Stefan Hajnoczi 2 participants 3 messages Aug 4

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gitlab: disable provenance attestations to work around CI bug

QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:

  $ docker push "$TAG"
  ...
  error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9

https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372

This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5

It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.

The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.

Cc Alex Bennée <[email protected]>
Cc Daniel P. Berrangé <[email protected]>
Cc Thomas Huth <[email protected]>

---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!

 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.d/container-template.yml +1 -1
index 8c7311cca57..88317d650e2 100644
@@ -14,7 +14,7 @@
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
     - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
+      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
       -f "tests/docker/dockerfiles/$NAME.docker" "."
     - docker push "$TAG"
   after_script:
Signature · 1 line
2.55.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Stefan Hajnoczi

QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:

  $ docker push "$TAG"
  ...
  error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
Quoted 8 lines

https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372

This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5

It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.

The forum post links to an issue where this comment:

https://github.com/moby/buildkit/issues/7007#issuecomment-5178990414

suggests we can just set

BUILDX_NO_DEFAULT_OCI_ARTIFACT=true

...except I don't think they've updated the docker docker image with this fix yet. Once they do, that will be better as we can set it in the GitLab env such that it applies to all branches, instead of havnig to cherry-pick

Quoted 25 lines

The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.

Cc Alex Bennée <[email protected]>
Cc Daniel P. Berrangé <[email protected]>
Cc Thomas Huth <[email protected]>

---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!

 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.d/container-template.yml +0 -0
index 8c7311cca57..88317d650e2 100644

.gitlab-ci.d/container-template.yml line 20

Quoted 8 lines
.gitlab-ci.d/container-template.yml +1 -1
@@ -14,7 +14,7 @@
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
     - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
+      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
       -f "tests/docker/dockerfiles/$NAME.docker" "."
     - docker push "$TAG"
   after_script:

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
author Applied
#2 lore raw
In reply to Stefan Hajnoczi
Quoted 28 lines · Stefan Hajnoczi

QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:

  $ docker push "$TAG"
  ...
  error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9

https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372

This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5

It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.

The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.

Cc Alex Bennée <[email protected]>
Cc Daniel P. Berrangé <[email protected]>
Cc Thomas Huth <[email protected]>
Quoted 7 lines

---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!

 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Thanks, applied to my staging tree:
https://gitlab.com/stefanha/qemu/commits/staging

Stefan