support/docker: generate reproducible base images
cover support/docker: generate reproducible base images
Hello All!
Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.
So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.
Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.
Regards,
Yann E. MORIN.
The following changes since commit 651af57cf011e6d8764d11932ce87fc064cd9a9b
dash: enable line editting if libedit is selected (2018-02-04 12:09:09 +0100)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to c7619a7bdc91c8c2cb9d486859ddad8a8e89e78a
support/dockker: create and push tagged images (2018-02-04 15:43:06 +0100)
----------------------------------------------------------------
Yann E. MORIN (5):
support/docker: limit the number of layers support/docker: don't be silent when setting up support/docker: use a known base distribution support/docker: use a fixed Debian snapshot support/dockker: create and push tagged images
support/docker/Dockerfile | 45 ++++++++++++++++++++++++----------------- support/docker/apt-sources.list | 4 ++++ 2 files changed, 30 insertions(+), 19 deletions(-) create mode 100644 support/docker/apt-sources.list
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/5 support/docker: limit the number of layers
The official documentation [0] suggests limiting the number of layers generated from a dockerfile. A layer is created for each RUN (and COPY and ADD) command. But we are only ever interested in the final image, so the intermediate layers are useless to us.
Limit the number of RUN commands to limit the number of generated layers.
--- support/docker/Dockerfile | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
support/docker/Dockerfile +15 -15
@@ -11,25 +11,25 @@ description="Container with everything needed to run Buildroot" # Setup environment ENV DEBIAN_FRONTEND noninteractive -RUN dpkg --add-architecture i386 # The container has no package lists, so need to update first -RUN apt-get update -y -qq -RUN apt-get install -y -qq --no-install-recommends \ - build-essential cmake libc6:i386 gcc-multilib \ - bc ca-certificates file locales rsync \ - cvs bzr git mercurial subversion wget \ - cpio unzip \ - libncurses5-dev \ - python-nose2 python-pexpect qemu-system-arm qemu-system-x86 -RUN apt-get -q -y autoremove -RUN apt-get -q -y clean +RUN dpkg --add-architecture i386 && \ + apt-get update -y -qq && \ + apt-get install -y -qq --no-install-recommends \ + build-essential cmake libc6:i386 gcc-multilib \ + bc ca-certificates file locales rsync \ + cvs bzr git mercurial subversion wget \ + cpio unzip \ + libncurses5-dev \ + python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \ + apt-get -q -y autoremove && \ + apt-get -q -y clean # To be able to generate a toolchain with locales, enable one UTF-8 locale -RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen -RUN /usr/sbin/locale-gen +RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \ + /usr/sbin/locale-gen -RUN useradd -ms /bin/bash br-user -RUN chown -R br-user:br-user /home/br-user +RUN useradd -ms /bin/bash br-user && \ + chown -R br-user:br-user /home/br-user USER br-user WORKDIR /home/br-user
Signature · 1 line
2.14.1
PATCH 2/5 support/docker: don't be silent when setting up
In reply to Yann E. MORINThis image is not built very often, and when it is, it is important to see what's going on, so don't be silent when installing packages from the distro, and since that can take a bit of time it thus serves as progress report...
--- support/docker/Dockerfile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
support/docker/Dockerfile +4 -4
@@ -13,16 +13,16 @@ ENV DEBIAN_FRONTEND noninteractive # The container has no package lists, so need to update first RUN dpkg --add-architecture i386 && \ - apt-get update -y -qq && \ - apt-get install -y -qq --no-install-recommends \ + apt-get update -y && \ + apt-get install -y --no-install-recommends \ build-essential cmake libc6:i386 gcc-multilib \ bc ca-certificates file locales rsync \ cvs bzr git mercurial subversion wget \ cpio unzip \ libncurses5-dev \ python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \ - apt-get -q -y autoremove && \ - apt-get -q -y clean + apt-get -y autoremove && \ + apt-get -y clean # To be able to generate a toolchain with locales, enable one UTF-8 locale RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
Signature · 1 line
2.14.1
PATCH 3/5 support/docker: use a known base distribution
In reply to Yann E. MORINCurrently, we are using debian:stable, which is subject to change with time, as new stable versions of Debian are released/updated.
Use the latest tagged stable release, stretch-20171210 as of today, as the base distribution to use.
This will ease reproducible builds in the future.
--- support/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
support/docker/Dockerfile +1 -1
@@ -3,7 +3,7 @@ # sudo docker build -t buildroot/base support/docker # sudo docker push buildroot/base -FROM debian:stable +FROM debian:stretch-20171210 LABEL maintainer="Buildroot mailing list <[email protected]>" \ vendor="Buildroot" \ description="Container with everything needed to run Buildroot"
Signature · 1 line
2.14.1
PATCH 4/5 support/docker: use a fixed Debian snapshot
In reply to Yann E. MORINSince we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.
As such, use the Debian snapshot that matches the base image.
--- support/docker/Dockerfile | 7 +++++++ support/docker/apt-sources.list | 4 ++++ 2 files changed, 11 insertions(+) create mode 100644 support/docker/apt-sources.list
support/docker/Dockerfile +7 -0
@@ -3,7 +3,11 @@ # sudo docker build -t buildroot/base support/docker # sudo docker push buildroot/base +# We use a specific tag for the base image *and* the corresponding date +# for the repository., so do not forget to update the apt-sources.list +# file that is shipped next to this Dockerfile. FROM debian:stretch-20171210 + LABEL maintainer="Buildroot mailing list <[email protected]>" \ vendor="Buildroot" \ description="Container with everything needed to run Buildroot"
@@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot" # Setup environment ENV DEBIAN_FRONTEND noninteractive +# This repository can be a bit slow at times. Don't panic... +COPY apt-sources.list /etc/apt/sources.list + # The container has no package lists, so need to update first RUN dpkg --add-architecture i386 && \ apt-get update -y && \
support/docker/apt-sources.list new file +4 -0
@@ -0,0 +1,4 @@ +# Latest just before 20171210T000000Z: +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
Signature · 1 line
2.14.1
PATCH 4/5 support/docker: use a fixed Debian snapshot
Quoted 47 lines · Yann E. MORIN
Since we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.
As such, use the Debian snapshot that matches the base image.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Peter Korsgaard <[email protected]>--- support/docker/Dockerfile | 7 +++++++ support/docker/apt-sources.list | 4 ++++ 2 files changed, 11 insertions(+) create mode 100644 support/docker/apt-sources.listsupport/docker/Dockerfile +7
-0@@ -3,7 +3,11 @@ # sudo docker build -t buildroot/base support/docker # sudo docker push buildroot/base +# We use a specific tag for the base image *and* the corresponding date +# for the repository., so do not forget to update the apt-sources.list +# file that is shipped next to this Dockerfile. FROM debian:stretch-20171210 + LABEL maintainer="Buildroot mailing list <[email protected]>" \ vendor="Buildroot" \ description="Container with everything needed to run Buildroot"@@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot" # Setup environment ENV DEBIAN_FRONTEND noninteractive +# This repository can be a bit slow at times. Don't panic... +COPY apt-sources.list /etc/apt/sources.list + # The container has no package lists, so need to update first RUN dpkg --add-architecture i386 && \ apt-get update -y && \support/docker/apt-sources.list new file +0
-0
support/docker/apt-sources.list line 2
Quoted 2 lines
support/docker/apt-sources.list new file +2
-0@@ -0,0 +1,4 @@ +# Latest just before 20171210T000000Z: +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main
AFAIU, snapshot should resolve any date to the latest available snapshot, so
http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well here. It should redirect to exactly the same URLs. Note thought that this doesn't work for the top-level, only for a specific release (because they are snapshotted at different times). So just clicking on the URL above won't work, but if you put dists/stretch after it it redirects to
http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/
Anyway, OK for now, but something to keep in mind when it is bumped.
Regards,
Arnout
Quoted 3 lines
support/docker/apt-sources.list new file +2
-0+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 4/5 support/docker: use a fixed Debian snapshot
Arnout, All,
On 2018-02-06 09:52 +0100, Arnout Vandecappelle spake thusly:
Quoted 4 lines · Yann E. MORIN
Quoted 3 lines
Since we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.
[--SNIP--]
Quoted 4 lines
Quoted 4 lines
support/docker/apt-sources.list new file +0
-1--- /dev/null
Quoted 8 lines
support/docker/apt-sources.list line 2
Quoted 4 lines
+++ b/support/docker/apt-sources.list
support/docker/apt-sources.list new file +2
-0@@ -0,0 +1,4 @@ +# Latest just before 20171210T000000Z: +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch mainAFAIU, snapshot should resolve any date to the latest available snapshot, so
http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well here. It should redirect to exactly the same URLs.
Yeah, I knew about the latest-prior-to-given-date rule, but see below...
Quoted 4 lines
Note thought that this
doesn't work for the top-level, only for a specific release (because they are snapshotted at different times). So just clicking on the URL above won't work, but if you put dists/stretch after iti
For me, I even got http 500, and apt-get update would also fail from time to time (about once for three).
That is indeed something I should have mentioned in the commit log.
Quoted 2 lines
it redirects to
http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/
nd what's even weird ios that it now works, even for the debian-security suite, which is not tagged at the same date:
http://snapshot.debian.org/archive/debian-security/20171209T224618Z/dists/stretch/updates/
Quoted 1 line
Anyway, OK for now, but something to keep in mind when it is bumped.
Yeah, I also would have preferred it, but as I said, it was sometimes failing for me... :-/
Thanks for the feedback! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 5/5 support/dockker: create and push tagged images
In reply to Yann E. MORINCurrently, we refer to the latest version of the image, which means we can't guarantee any reproducibility. Also, it measn we can't have a separate images for the maintenance branches (especially the LTS) and master.
Update the comment in the Dockerfile to create and push tagged images.
---
Note: once this is applied, we can push such a tagged image and then update our .gitlab-ci.yml accordingly.
--- support/docker/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
support/docker/Dockerfile +3 -3
@@ -1,7 +1,7 @@ # This Dockerfile generates the docker image that gets used by Gitlab CI -# To build it: -# sudo docker build -t buildroot/base support/docker -# sudo docker push buildroot/base +# To build it (YYYYMMDD.HHMM is the current date and time in UTC): +# sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker +# sudo docker push buildroot/base:YYYYMMDD.HHMM # We use a specific tag for the base image *and* the corresponding date # for the repository., so do not forget to update the apt-sources.list
Signature · 1 line
2.14.1
cover support/docker: generate reproducible base images
In reply to Yann E. MORINQuoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hello All!
Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.
Quoted 5 lines
So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.
Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.
Committed series, thanks - You are now officially our Docker expert ;)
Signature · 1 line
Bye, Peter Korsgaard
cover support/docker: generate reproducible base images
Peter, All,
On 2018-02-04 22:00 +0100, Peter Korsgaard spake thusly:
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.
So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.
Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.
Quoted 1 line
Committed series, thanks - You are now officially our Docker expert ;)
Oh, now, we do have Cam Hutchison as an expert! ;-)
I'll regenerate the image now, push it, and update our .gitlab-ci.yml now. Expect patch any minute now...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'