Skip to contentSkip to conversation

support/docker: generate reproducible base images

PATCH New Yann E. MORIN 3 participants 10 messages Feb 4, 2018 – Feb 6, 2018

lore
Share & tools
lore
#0 lore raw

cover support/docker: generate reproducible base images

Hello All!

Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.

So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.

Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.

Regards,
Yann E. MORIN.

The following changes since commit 651af57cf011e6d8764d11932ce87fc064cd9a9b

dash: enable line editting if libedit is selected (2018-02-04 12:09:09 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to c7619a7bdc91c8c2cb9d486859ddad8a8e89e78a

support/dockker: create and push tagged images (2018-02-04 15:43:06 +0100)

----------------------------------------------------------------
Yann E. MORIN (5):
support/docker: limit the number of layers support/docker: don't be silent when setting up support/docker: use a known base distribution support/docker: use a fixed Debian snapshot support/dockker: create and push tagged images

 support/docker/Dockerfile       | 45 ++++++++++++++++++++++++-----------------
 support/docker/apt-sources.list |  4 ++++
 2 files changed, 30 insertions(+), 19 deletions(-)
 create mode 100644 support/docker/apt-sources.list
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/5 support/docker: limit the number of layers

The official documentation [0] suggests limiting the number of layers generated from a dockerfile. A layer is created for each RUN (and COPY and ADD) command. But we are only ever interested in the final image, so the intermediate layers are useless to us.

Limit the number of RUN commands to limit the number of generated layers.

[0] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#minimize-the-number-of-layers

Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 support/docker/Dockerfile | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
support/docker/Dockerfile +15 -15
index ebb471f7e5..23018585eb 100644
@@ -11,25 +11,25 @@ description="Container with everything needed to run Buildroot"
 # Setup environment
 ENV DEBIAN_FRONTEND noninteractive
 
-RUN dpkg --add-architecture i386
 # The container has no package lists, so need to update first
-RUN apt-get update -y -qq
-RUN apt-get install -y -qq --no-install-recommends \
-    build-essential cmake libc6:i386 gcc-multilib \
-    bc ca-certificates file locales rsync \
-    cvs bzr git mercurial subversion wget \
-    cpio unzip \
-    libncurses5-dev \
-    python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-RUN apt-get -q -y autoremove
-RUN apt-get -q -y clean
+RUN dpkg --add-architecture i386 && \
+    apt-get update -y -qq && \
+    apt-get install -y -qq --no-install-recommends \
+        build-essential cmake libc6:i386 gcc-multilib \
+        bc ca-certificates file locales rsync \
+        cvs bzr git mercurial subversion wget \
+        cpio unzip \
+        libncurses5-dev \
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \
+    apt-get -q -y autoremove && \
+    apt-get -q -y clean
 
 # To be able to generate a toolchain with locales, enable one UTF-8 locale
-RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-RUN /usr/sbin/locale-gen
+RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
+    /usr/sbin/locale-gen
 
-RUN useradd -ms /bin/bash br-user
-RUN chown -R br-user:br-user /home/br-user
+RUN useradd -ms /bin/bash br-user && \
+    chown -R br-user:br-user /home/br-user
 
 USER br-user
 WORKDIR /home/br-user
Signature · 1 line
2.14.1
#2 lore raw

PATCH 2/5 support/docker: don't be silent when setting up

In reply to Yann E. MORIN

This image is not built very often, and when it is, it is important to see what's going on, so don't be silent when installing packages from the distro, and since that can take a bit of time it thus serves as progress report...

Cc Arnout Vandecappelle <[email protected]>
---
 support/docker/Dockerfile | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
support/docker/Dockerfile +4 -4
index 23018585eb..aa2b38fcc9 100644
@@ -13,16 +13,16 @@ ENV DEBIAN_FRONTEND noninteractive
 
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
-    apt-get update -y -qq && \
-    apt-get install -y -qq --no-install-recommends \
+    apt-get update -y && \
+    apt-get install -y --no-install-recommends \
         build-essential cmake libc6:i386 gcc-multilib \
         bc ca-certificates file locales rsync \
         cvs bzr git mercurial subversion wget \
         cpio unzip \
         libncurses5-dev \
         python-nose2 python-pexpect qemu-system-arm qemu-system-x86 && \
-    apt-get -q -y autoremove && \
-    apt-get -q -y clean
+    apt-get -y autoremove && \
+    apt-get -y clean
 
 # To be able to generate a toolchain with locales, enable one UTF-8 locale
 RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
Signature · 1 line
2.14.1
#3 lore raw

PATCH 3/5 support/docker: use a known base distribution

In reply to Yann E. MORIN

Currently, we are using debian:stable, which is subject to change with time, as new stable versions of Debian are released/updated.

Use the latest tagged stable release, stretch-20171210 as of today, as the base distribution to use.

This will ease reproducible builds in the future.

Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 support/docker/Dockerfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/docker/Dockerfile +1 -1
index aa2b38fcc9..49b74b88bd 100644
@@ -3,7 +3,7 @@
 #   sudo docker build -t buildroot/base support/docker
 #   sudo docker push buildroot/base
 
-FROM debian:stable
+FROM debian:stretch-20171210
 LABEL maintainer="Buildroot mailing list <[email protected]>" \
       vendor="Buildroot" \
 description="Container with everything needed to run Buildroot"
Signature · 1 line
2.14.1
#4 lore raw

PATCH 4/5 support/docker: use a fixed Debian snapshot

In reply to Yann E. MORIN

Since we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.

As such, use the Debian snapshot that matches the base image.

Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 support/docker/Dockerfile       | 7 +++++++
 support/docker/apt-sources.list | 4 ++++
 2 files changed, 11 insertions(+)
 create mode 100644 support/docker/apt-sources.list
support/docker/Dockerfile +7 -0
index 49b74b88bd..f18d195f37 100644
@@ -3,7 +3,11 @@
 #   sudo docker build -t buildroot/base support/docker
 #   sudo docker push buildroot/base
 
+# We use a specific tag for the base image *and* the corresponding date
+# for the repository., so do not forget to update the apt-sources.list
+# file that is shipped next to this Dockerfile.
 FROM debian:stretch-20171210
+
 LABEL maintainer="Buildroot mailing list <[email protected]>" \
       vendor="Buildroot" \
 description="Container with everything needed to run Buildroot"
@@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot"
 # Setup environment
 ENV DEBIAN_FRONTEND noninteractive
 
+# This repository can be a bit slow at times. Don't panic...
+COPY apt-sources.list /etc/apt/sources.list
+
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
     apt-get update -y && \
support/docker/apt-sources.list new file +4 -0
new file mode 100644
index 0000000000..789fb8fc17
@@ -0,0 +1,4 @@
+# Latest just before 20171210T000000Z:
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
Signature · 1 line
2.14.1
#5 lore raw

PATCH 4/5 support/docker: use a fixed Debian snapshot

Quoted 47 lines · Yann E. MORIN

Since we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.

As such, use the Debian snapshot that matches the base image.

Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 support/docker/Dockerfile       | 7 +++++++
 support/docker/apt-sources.list | 4 ++++
 2 files changed, 11 insertions(+)
 create mode 100644 support/docker/apt-sources.list
support/docker/Dockerfile +7 -0
index 49b74b88bd..f18d195f37 100644
@@ -3,7 +3,11 @@
 #   sudo docker build -t buildroot/base support/docker
 #   sudo docker push buildroot/base
 
+# We use a specific tag for the base image *and* the corresponding date
+# for the repository., so do not forget to update the apt-sources.list
+# file that is shipped next to this Dockerfile.
 FROM debian:stretch-20171210
+
 LABEL maintainer="Buildroot mailing list <[email protected]>" \
       vendor="Buildroot" \
 description="Container with everything needed to run Buildroot"
@@ -11,6 +15,9 @@ description="Container with everything needed to run Buildroot"
 # Setup environment
 ENV DEBIAN_FRONTEND noninteractive
 
+# This repository can be a bit slow at times. Don't panic...
+COPY apt-sources.list /etc/apt/sources.list
+
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
     apt-get update -y && \
support/docker/apt-sources.list new file +0 -0
new file mode 100644
index 0000000000..789fb8fc17

support/docker/apt-sources.list line 2

Quoted 2 lines
support/docker/apt-sources.list new file +2 -0
@@ -0,0 +1,4 @@
+# Latest just before 20171210T000000Z:
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main

AFAIU, snapshot should resolve any date to the latest available snapshot, so
http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well here. It should redirect to exactly the same URLs. Note thought that this doesn't work for the top-level, only for a specific release (because they are snapshotted at different times). So just clicking on the URL above won't work, but if you put dists/stretch after it it redirects to
http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/

Anyway, OK for now, but something to keep in mind when it is bumped.

Regards,
Arnout

Quoted 3 lines
support/docker/apt-sources.list new file +2 -0
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#6 lore raw

PATCH 4/5 support/docker: use a fixed Debian snapshot

Arnout, All,

On 2018-02-06 09:52 +0100, Arnout Vandecappelle spake thusly:

Quoted 4 lines · Yann E. MORIN
Quoted 3 lines

Since we're now using a specific base image tag, we need to also use a specific, stable repository to get additional packages from for this image.

[--SNIP--]

Quoted 4 lines
Quoted 4 lines
support/docker/apt-sources.list new file +0 -1
new file mode 100644
index 0000000000..789fb8fc17
--- /dev/null
Quoted 8 lines

support/docker/apt-sources.list line 2

Quoted 4 lines

+++ b/support/docker/apt-sources.list

support/docker/apt-sources.list new file +2 -0
@@ -0,0 +1,4 @@
+# Latest just before 20171210T000000Z:
+deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main

AFAIU, snapshot should resolve any date to the latest available snapshot, so
http://snapshot.debian.org/archive/debian/20171210T000000Z/ should work as well here. It should redirect to exactly the same URLs.

Yeah, I knew about the latest-prior-to-given-date rule, but see below...

Quoted 4 lines

Note thought that this
doesn't work for the top-level, only for a specific release (because they are snapshotted at different times). So just clicking on the URL above won't work, but if you put dists/stretch after iti

For me, I even got http 500, and apt-get update would also fail from time to time (about once for three).

That is indeed something I should have mentioned in the commit log.

Quoted 2 lines

it redirects to
http://snapshot.debian.org/archive/debian/20171209T220346Z/dists/stretch/

nd what's even weird ios that it now works, even for the debian-security suite, which is not tagged at the same date:

    http://snapshot.debian.org/archive/debian-security/20171209T224618Z/dists/stretch/updates/
Quoted 1 line

Anyway, OK for now, but something to keep in mind when it is bumped.

Yeah, I also would have preferred it, but as I said, it was sometimes failing for me... :-/

Thanks for the feedback! :-)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH 5/5 support/dockker: create and push tagged images

In reply to Yann E. MORIN

Currently, we refer to the latest version of the image, which means we can't guarantee any reproducibility. Also, it measn we can't have a separate images for the maintenance branches (especially the LTS) and master.

Update the comment in the Dockerfile to create and push tagged images.

Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>

---
Note: once this is applied, we can push such a tagged image and then update our .gitlab-ci.yml accordingly.

---
 support/docker/Dockerfile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
support/docker/Dockerfile +3 -3
index f18d195f37..474e073c61 100644
@@ -1,7 +1,7 @@
 # This Dockerfile generates the docker image that gets used by Gitlab CI
-# To build it:
-#   sudo docker build -t buildroot/base support/docker
-#   sudo docker push buildroot/base
+# To build it (YYYYMMDD.HHMM is the current date and time in UTC):
+#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker
+#   sudo docker push buildroot/base:YYYYMMDD.HHMM
 
 # We use a specific tag for the base image *and* the corresponding date
 # for the repository., so do not forget to update the apt-sources.list
Signature · 1 line
2.14.1
#8 lore raw

cover support/docker: generate reproducible base images

In reply to Yann E. MORIN
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Hello All!
Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.

Quoted 5 lines

So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.

Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.

Committed series, thanks - You are now officially our Docker expert ;)

Signature · 1 line
Bye, Peter Korsgaard
#9 lore raw

cover support/docker: generate reproducible base images

Peter, All,

On 2018-02-04 22:00 +0100, Peter Korsgaard spake thusly:

Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Currently, we only ever refer to the "latest" version of the image. This is not nice because:
1. this is not reproducible;
2. we can't have different images between master and the various maintenance branches.
So, this series makes the generated image reproducible, and explains how to gnerate and push a tagged image.
Once this series has been applied, we'll have to generate such a tagged image and update our .gitlab-ci.yml accordingly.

Quoted 1 line

Committed series, thanks - You are now officially our Docker expert ;)

Oh, now, we do have Cam Hutchison as an expert! ;-)

I'll regenerate the image now, push it, and update our .gitlab-ci.yml now. Expect patch any minute now...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'