support/docker: make the image more reproducible

Hello All!

Currently, the base test image is not entirely reproducible, because we install packages from PyPI without specifying all the versions of all the packages we install from there.

Instead of relying on a third-party to provide packages, just use those from the distribution, because we are using a pinned version and thus we will always get the same packages from the distro.

If/when this series is applied, I'll regenerate and updalod the new buildroot/base image, and send a furhter patch to use it from our gitlab-ci.yml.

Regards,
Yann E. MORIN.

The following changes since commit 12b70acb82acfa4925a8436a31a668ac92a58d1d

mender: fix check-package warnings (2018-06-10 16:43:53 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 720ecb41c6c51f66f6b4dda7663c1e16edb2fabb

support/docker: use the distro-provided flake8 (2018-06-10 17:41:00 +0200)

----------------------------------------------------------------
Yann E. MORIN (2):
support/docker: sort the list of installed packages support/docker: use the distro-provided flake8

 support/docker/Dockerfile | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

As suggested in the docker best practices [0], order the package list alphabetically, and list only one package per line.

This will be much usefull later, we need to update the list of installed packages, like adding new ones for example.

[0] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#sort-multi-line-arguments

---
Changes v2 -> v3:
- don't mistakenly drop cmake (Ricardo)

Changes v1 -> v2:
- don't drop python-pip yet (Ricardo)

---
 support/docker/Dockerfile | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

index 8c525f7cf1..bebd861dc1 100644
@@ -22,13 +22,30 @@ COPY apt-sources.list /etc/apt/sources.list
 RUN dpkg --add-architecture i386 && \
     apt-get update -y
 RUN apt-get install -y --no-install-recommends \
-        build-essential cmake libc6:i386 g++-multilib \
-        bc ca-certificates file locales rsync \
-        cvs bzr git mercurial subversion wget \
-        cpio unzip \
+        bc \
+        build-essential \
+        bzr \
+        ca-certificates \
+        cmake \
+        cpio \
+        cvs \
+        file \
+        g++-multilib \
+        git \
+        libc6:i386 \
         libncurses5-dev \
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86 \
-        python-pip && \
+        locales \
+        mercurial \
+        python-nose2 \
+        python-pexpect \
+        python-pip \
+        qemu-system-arm \
+        qemu-system-x86 \
+        rsync \
+        subversion \
+        unzip \
+        wget \
+        && \
     apt-get -y autoremove && \
     apt-get -y clean
 

2.14.1

Hello,

Regards,
Ricardo

Currently, we install flake8 and its dependencies via pip. We tried to be reproducible by pinning the version of those python packages, but we did forget quite a few of them, and thus some dependencies for flake8 are installed as uncontrolled versions.

Furthermore, before we install flake8 and its dependencies, we forcibly update pip, setuptools, and wheels packages to their latest versions. This explicitly breaks reproducibility.

While we could enforce a specific version of all those packages and still grab them from PyPI, we can simply grab them from the distribution-provided packages instead.

Since we're using a pinned version of stretch, this already guarantees we'll reproducibly get the same versions over and over again. Besides, we just need to list flake8 as a package to install to automatically get all its dependencies (again, in a reproducible way).

This has the slight unfortunate drawback of downgrading flake8 to version 3.2.1, from version 3.5.0, as well as downgrading a few of flake8's dependencies, as noticed by Ricardo:

    http://lists.busybox.net/pipermail/buildroot/2018-May/222376.html

However, as Ricardo said, there isn't "any serious limitation of this old version, the release notes for a version in the between mentions 'Dramatically improve the performance' but we have a limited number of scripts and running on Gitlab for all of them still takes less than 5 minutes".

---
Changes v1 -> v2:
- improve commit log to mention downgrading of flake8 (Ricardo)
- correctly use python2's version of flake8, not python3's (Ricardo)
- fix typoes (Ricardo)

---
 support/docker/Dockerfile | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

index bebd861dc1..36cd3b28a4 100644
@@ -36,9 +36,9 @@ RUN apt-get install -y --no-install-recommends \
         libncurses5-dev \
         locales \
         mercurial \
+        python-flake8 \
         python-nose2 \
         python-pexpect \
-        python-pip \
         qemu-system-arm \
         qemu-system-x86 \
         rsync \

@@ -49,14 +49,6 @@ RUN apt-get install -y --no-install-recommends \
     apt-get -y autoremove && \
     apt-get -y clean
 
-# For check-flake8
-RUN python -m pip install --upgrade pip setuptools wheel && \
-    pip install -q \
-        flake8==3.5.0 \
-        mccabe==0.6.1 \
-        pycodestyle==2.3.1 \
-        pyflakes==1.6.0
-
 # To be able to generate a toolchain with locales, enable one UTF-8 locale
 RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
     /usr/sbin/locale-gen

2.14.1

Hello,

Both applied, thanks!

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com