gitlab: disable provenance attestations to work around CI bug
PATCH 1/1 gitlab: disable provenance attestations to work around CI bug
QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:
$ docker push "$TAG" ... error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.
The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.
---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!
.gitlab-ci.d/container-template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.d/container-template.yml +1 -1
@@ -14,7 +14,7 @@ - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" - --build-arg BUILDKIT_INLINE_CACHE=1 + --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false -f "tests/docker/dockerfiles/$NAME.docker" "." - docker push "$TAG" after_script:
Signature · 1 line
2.55.0
Quoted 8 lines · Stefan Hajnoczi
QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:
$ docker push "$TAG" ... error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
Quoted 8 lines
https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.
The forum post links to an issue where this comment:
https://github.com/moby/buildkit/issues/7007#issuecomment-5178990414
suggests we can just set
BUILDX_NO_DEFAULT_OCI_ARTIFACT=true
...except I don't think they've updated the docker docker image with this fix yet. Once they do, that will be better as we can set it in the GitLab env such that it applies to all branches, instead of havnig to cherry-pick
Quoted 25 lines
The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.
Cc Alex Bennée <[email protected]>Cc Daniel P. Berrangé <[email protected]>Cc Thomas Huth <[email protected]>Signed-off-by Stefan Hajnoczi <[email protected]>---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!.gitlab-ci.d/container-template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-).gitlab-ci.d/container-template.yml +0
-0
.gitlab-ci.d/container-template.yml line 20
Quoted 8 lines
.gitlab-ci.d/container-template.yml +1
-1@@ -14,7 +14,7 @@ - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" - --build-arg BUILDKIT_INLINE_CACHE=1 + --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false -f "tests/docker/dockerfiles/$NAME.docker" "." - docker push "$TAG" after_script:
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
Stefan Hajnoczi
AppliedQuoted 28 lines · Stefan Hajnoczi
QEMU's CI pipeline involves building container images that will be used to run builds and tests. A recent Docker change triggered the following error:
$ docker push "$TAG" ... error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
This happens because Docker now pushes out-of-order and the GitLab Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5It is unclear at this point whether GitLab will modify the behavior of Container Registry or whether Docker will ship a fix.
The current workaround is to disable the provenance attestation that is involved in this issue. QEMU's CI pipeline container images are used internally for testing and are not widely distributed. Provenance attestation can be disabled as there are no external consumers of these images. Expect to revert this commit in the future when GitLab or Docker have released their own fixes.
Cc Alex Bennée <[email protected]>Cc Daniel P. Berrangé <[email protected]>Cc Thomas Huth <[email protected]>Signed-off-by Stefan Hajnoczi <[email protected]>
Quoted 7 lines
---
Note that this is an urgent patch so that the CI will pass again and I can tag v11.1.0-rc3 later today. Currently it is failing due to this issue!.gitlab-ci.d/container-template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Thanks, applied to my staging tree:
https://gitlab.com/stefanha/qemu/commits/staging
Stefan