Skip to contentSkip to conversation

support/docker: add new dockers (branch yem/dockers)

PATCH New Yann E. MORIN 3 participants 11 messages May 17, 2020 – Jun 14, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover support/docker: add new dockers (branch yem/dockers)

Hello All!

Enhance our docker support, update the existing image, add a new one.

This will help:

- users, to get a build environment known to work,

- developpers, to more easily reproduce build failures reported by users.

A basic set of a few, very various distros have been chosen, to cover a range of diversity:

- the existing, rock-solid Debian Stretch, pinned to a known state,

    that we use for the Gitlab-CI testing;

- Fedora-F32, a bleeding-edge mainstream distro, pinned to the state

    it was at release time;

More variants can be added later, depending on the demand.

Note: when this series is applied (if it is), I'll generate the images and push them to the Buildroot repo on the Docker hub.

Changes v1 -> v2:
- reorder patches to ensure bisectability (Ricardo)
- drop the Makefile rule in favour of a script
- drop Arch and Alpine for now...

Regards,
Yann E. MORIN.

The following changes since commit 53158e41fdd4bcc1272bbf4e61b68f58bf9c1527

package/openocd: fix -fno-common build failure (2020-05-17 21:50:04 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 48c45cb866a160ce4cc05b03fdf15237eb346b66

support/docker: add a Fedore-32 based image (2020-05-17 23:31:36 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
support/docker: rename the Dockerfile and its accompanying file support/docker: add script to generate docker iamges support/docker: allow the br-user to becone root support/docker: add a Fedore-32 based image

 support/docker/Dockerfile.base                     |  1 +
 .../{Dockerfile => Dockerfile.debian-stretch}      | 12 ++++--
 support/docker/Dockerfile.fedora-32                | 48 ++++++++++++++++++++++
 support/docker/build-all                           | 17 ++++++++
 ...ources.list => debian-stretch.apt-sources.list} |  0
 5 files changed, 75 insertions(+), 3 deletions(-)
 create mode 120000 support/docker/Dockerfile.base
 rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (80%)
 create mode 100644 support/docker/Dockerfile.fedora-32
 create mode 100755 support/docker/build-all
 rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v2 1/4 support/docker: rename the Dockerfile and its accompanying file

Currently, we only generate one docker image, for use by the gitlab-ci infra.

However, users have expressed an interest in using that image for other uses, like:

- reproducing build failures as reported by the gitlab-ci jobs;

- having a ready-to-use base image to use as-is, or as a template for further customisations.

Additionally, some users have reported the need for similar base images, but based off different distros, with the same goals as above, plus:

- reproducing build failures reported by users on other distros.

As a first step toward providing such images, rename our Dockerfile to include the name and version of the distro it is based off.

We add a symlink so that we still have a 'base' image that we can reference from the gitlab-ci infra.

Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/docker/Dockerfile.base                             | 1 +
 support/docker/{Dockerfile => Dockerfile.debian-stretch}   | 7 +++++--
 .../{apt-sources.list => debian-stretch.apt-sources.list}  | 0
 3 files changed, 6 insertions(+), 2 deletions(-)
 create mode 120000 support/docker/Dockerfile.base
 rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (88%)
 rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)
support/docker/Dockerfile.base new file +1 -0
new file mode 120000
index 0000000000..06a988b45e
@@ -0,0 +1 @@
+Dockerfile.debian-stretch
\ No newline at end of file
support/docker/Dockerfile.debian-stretch renamed from support/docker/Dockerfile +5 -2
similarity index 88%
rename from support/docker/Dockerfile
rename to support/docker/Dockerfile.debian-stretch
index 03acde85d2..0d2ce2aa79 100644
@@ -1,6 +1,9 @@
+# vi: ft=dockerfile
 # This Dockerfile generates the docker image that gets used by Gitlab CI
 # To build it (YYYYMMDD.HHMM is the current date and time in UTC):
-#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker
+#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM \
+#                     -f support/docker/Dockerfile.debian-stretch \
+#                     support/docker
 #   sudo docker push buildroot/base:YYYYMMDD.HHMM
 
 # We use a specific tag for the base image *and* the corresponding date
@@ -16,7 +19,7 @@ description="Container with everything needed to run Buildroot"
 ENV DEBIAN_FRONTEND noninteractive
 
 # This repository can be a bit slow at times. Don't panic...
-COPY apt-sources.list /etc/apt/sources.list
+COPY debian-stretch.apt-sources.list /etc/apt/sources.list
 
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
support/docker/debian-stretch.apt-sources.list renamed from support/docker/apt-sources.list +0 -0
similarity index 100%
rename from support/docker/apt-sources.list
rename to support/docker/debian-stretch.apt-sources.list
Signature · 1 line
2.20.1
#2 lore raw

PATCH v2 1/4 support/docker: rename the Dockerfile and its accompanying file

Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:

Quoted 19 lines

Currently, we only generate one docker image, for use by the gitlab-ci infra.

However, users have expressed an interest in using that image for other uses, like:

- reproducing build failures as reported by the gitlab-ci jobs;

- having a ready-to-use base image to use as-is, or as a template for further customisations.

Additionally, some users have reported the need for similar base images, but based off different distros, with the same goals as above, plus:

- reproducing build failures reported by users on other distros.

As a first step toward providing such images, rename our Dockerfile to include the name and version of the distro it is based off.

Quoted 6 lines

We add a symlink so that we still have a 'base' image that we can reference from the gitlab-ci infra.

Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Best regards,
Romain

Quoted 49 lines
---
 support/docker/Dockerfile.base                             | 1 +
 support/docker/{Dockerfile => Dockerfile.debian-stretch}   | 7 +++++--
 .../{apt-sources.list => debian-stretch.apt-sources.list}  | 0
 3 files changed, 6 insertions(+), 2 deletions(-)
 create mode 120000 support/docker/Dockerfile.base
 rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (88%)
 rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)
support/docker/Dockerfile.base new file +1 -0
new file mode 120000
index 0000000000..06a988b45e
@@ -0,0 +1 @@
+Dockerfile.debian-stretch
\ No newline at end of file
support/docker/Dockerfile.debian-stretch renamed from support/docker/Dockerfile +5 -2
similarity index 88%
rename from support/docker/Dockerfile
rename to support/docker/Dockerfile.debian-stretch
index 03acde85d2..0d2ce2aa79 100644
@@ -1,6 +1,9 @@
+# vi: ft=dockerfile
 # This Dockerfile generates the docker image that gets used by Gitlab CI
 # To build it (YYYYMMDD.HHMM is the current date and time in UTC):
-#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker
+#   sudo docker build -t buildroot/base:YYYYMMDD.HHMM \
+#                     -f support/docker/Dockerfile.debian-stretch \
+#                     support/docker
 #   sudo docker push buildroot/base:YYYYMMDD.HHMM
 
 # We use a specific tag for the base image *and* the corresponding date
@@ -16,7 +19,7 @@ description="Container with everything needed to run Buildroot"
 ENV DEBIAN_FRONTEND noninteractive
 
 # This repository can be a bit slow at times. Don't panic...
-COPY apt-sources.list /etc/apt/sources.list
+COPY debian-stretch.apt-sources.list /etc/apt/sources.list
 
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
support/docker/debian-stretch.apt-sources.list renamed from support/docker/apt-sources.list +0 -0
similarity index 100%
rename from support/docker/apt-sources.list
rename to support/docker/debian-stretch.apt-sources.list
#3 lore raw

PATCH v2 2/4 support/docker: add script to generate docker iamges

In reply to Yann E. MORIN
Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/docker/build-all | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100755 support/docker/build-all
support/docker/build-all new file +17 -0
new file mode 100755
index 0000000000..4f3217e7fe
@@ -0,0 +1,17 @@
+#!/bin/sh
+set -e
+
+BR2_DOCKER_VERSION="${1}"
+
+if [ -z "${BR2_DOCKER_VERSION}" ]; then
+    printf "Usage: ${0} VERSION\n"
+    exit 1
+fi
+
+CONTEXT_DIR="${0%/*}"
+for d in "${CONTEXT_DIR}"/Dockerfile.*; do \
+    distro="${d#*/Dockerfile.}"
+    docker build \
+        -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \
+        -f "${d}" "${CONTEXT_DIR}"
+done
Signature · 1 line
2.20.1
#4 lore raw

PATCH v2 2/4 support/docker: add script to generate docker iamges

Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:

Quoted 14 lines
Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 support/docker/build-all | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100755 support/docker/build-all
support/docker/build-all new file +0 -0
new file mode 100755
index 0000000000..4f3217e7fe

support/docker/build-all line 2

Quoted 2 lines
support/docker/build-all new file +2 -0
@@ -0,0 +1,17 @@
+#!/bin/sh
+set -e

Best regards,
Romain

Quoted 16 lines
support/docker/build-all new file +15 -0
+
+BR2_DOCKER_VERSION="${1}"
+
+if [ -z "${BR2_DOCKER_VERSION}" ]; then
+    printf "Usage: ${0} VERSION\n"
+    exit 1
+fi
+
+CONTEXT_DIR="${0%/*}"
+for d in "${CONTEXT_DIR}"/Dockerfile.*; do \
+    distro="${d#*/Dockerfile.}"
+    docker build \
+        -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \
+        -f "${d}" "${CONTEXT_DIR}"
+done
#5 lore raw

PATCH v2 3/4 support/docker: allow the br-user to becone root

In reply to Yann E. MORIN

When our Dockerfiles (and images) get reused by users, they might need to locally add new packages or tweak the iamge to heir local needs.

As such, register the br-user to become root by running (exactly!):

    $ sudo su -

This will allow acutally becoing root without prompting for a password, but no other command will be accepted, and no other paramter will be accepted either.

Cc; Thomas Petazzoni <[email protected]>

---
 support/docker/Dockerfile.debian-stretch | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
support/docker/Dockerfile.debian-stretch +4 -1
index 0d2ce2aa79..7ce9f406c6 100644
@@ -49,6 +49,7 @@ RUN apt-get install -y --no-install-recommends \
         qemu-system-x86 \
         rsync \
         subversion \
+        sudo \
         unzip \
         wget \
         && \
@@ -60,7 +61,9 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
     /usr/sbin/locale-gen
 
 RUN useradd -ms /bin/bash br-user && \
-    chown -R br-user:br-user /home/br-user
+    chown -R br-user:br-user /home/br-user && \
+    printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
 
 USER br-user
 WORKDIR /home/br-user
Signature · 1 line
2.20.1
#6 lore raw

PATCH v2 3/4 support/docker: allow the br-user to becone root

Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:

Quoted 4 lines

When our Dockerfiles (and images) get reused by users, they might need to locally add new packages or tweak the iamge to heir local needs.

As such, register the br-user to become root by running (exactly!):

Quoted 8 lines
    $ sudo su -

This will allow acutally becoing root without prompting for a password, but no other command will be accepted, and no other paramter will be accepted either.

Cc; Thomas Petazzoni <[email protected]>

Best regards,
Romain

Quoted 28 lines
---
 support/docker/Dockerfile.debian-stretch | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
support/docker/Dockerfile.debian-stretch +4 -1
index 0d2ce2aa79..7ce9f406c6 100644
@@ -49,6 +49,7 @@ RUN apt-get install -y --no-install-recommends \
         qemu-system-x86 \
         rsync \
         subversion \
+        sudo \
         unzip \
         wget \
         && \
@@ -60,7 +61,9 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
     /usr/sbin/locale-gen
 
 RUN useradd -ms /bin/bash br-user && \
-    chown -R br-user:br-user /home/br-user
+    chown -R br-user:br-user /home/br-user && \
+    printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
 
 USER br-user
 WORKDIR /home/br-user
#7 lore raw

PATCH v2 4/4 support/docker: add a Fedore-32 based image

In reply to Yann E. MORIN

Hopefully, this image is reproducible, since we are not using any update repository, and from the looks of it, the repository of the release is never touched.

Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>

---
Changes v1 -> v2:
- fix ordering of packages list (Ricardo)

---
 support/docker/Dockerfile.fedora-32 | 48 +++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 support/docker/Dockerfile.fedora-32
support/docker/Dockerfile.fedora-32 new file +48 -0
new file mode 100644
index 0000000000..85e5c4ba9c
@@ -0,0 +1,48 @@
+# vi: ft=dockerfile
+# Note: this dockerfile is not reproducible, because the repository
+# where we fetch packages from is not pinned to a specific point in
+# time; i.e. we'll get whatever is current at the moment we build.
+
+FROM fedora:32
+
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+      description="Container based on Fedora 32, with everything needed to run Buildroot already installed"
+
+# Remove the repository with updates, so we only
+# get packages as they were at time of release.
+RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \
+    dnf -y install \
+        @development-tools \
+        bc \
+        bzip2 \
+        bzr \
+        cpio \
+        cvs \
+        file \
+        gcc-c++ \
+        git \
+        mercurial \
+        ncurses-devel \
+        perl-ExtUtils-MakeMaker \
+        perl-Thread-Queue \
+        python-nose2 \
+        python-pexpect \
+        qemu-system-arm \
+        qemu-system-x86 \
+        rsync \
+        subversion \
+        unzip \
+        wget \
+        which \
+    && dnf -y clean all
+
+RUN useradd -ms /bin/bash br-user && \
+    chown -R br-user:br-user /home/br-user && \
+    printf 'br-user ALL=NOPASSWD: /usr/bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
+
+USER br-user
+WORKDIR /home/br-user
+ENV HOME /home/br-user
+ENV LC_ALL C.utf8
Signature · 1 line
2.20.1
#8 lore raw

PATCH v2 4/4 support/docker: add a Fedore-32 based image

Hello,

Thanks for doing this work, I did need some Docker images just yesterday. One comment/question below.

Quoted 11 lines · On Sun, 17 May 2020 23:41:40 +0200 "Yann E. MORIN" <[email protected]>…
support/docker/Dockerfile.fedora-32 new file +11 -0
+# Remove the repository with updates, so we only
+# get packages as they were at time of release.
+RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \
+    dnf -y install \
+        @development-tools \
+        bc \
+        bzip2 \
+        bzr \
+        cpio \
+        cvs \
+        file \

support/docker/Dockerfile.fedora-32 line 32

Quoted 10 lines
support/docker/Dockerfile.fedora-32 new file +10 -0
+        gcc-c++ \
+        git \
+        mercurial \
+        ncurses-devel \
+        perl-ExtUtils-MakeMaker \
+        perl-Thread-Queue \
+        python-nose2 \
+        python-pexpect \
+        qemu-system-arm \
+        qemu-system-x86 \

With python-nose2 and python-pexpect, this pulls in a Python interpreter, which is not strictly a requirement of Buildroot. I know these are needed to run the runtime tests, and therefore to run the Gitlab CI stuff. But sometimes we need to do some tests with really the minimal set of Buildroot dependencies. Same goes with qemu-system-* packages.

Should we have two sets of Docker images, one with the minimal set of dependencies, and one that adds the stuff needed for Gitlab CI ? I believe Docker allows to create some kind of layered "thing" where an image sits on top of another.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#9 lore raw

PATCH v2 4/4 support/docker: add a Fedore-32 based image

In reply to Yann E. MORIN

Commit tittle : Fedore/Fedora

Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:

Quoted 35 lines

Hopefully, this image is reproducible, since we are not using any update repository, and from the looks of it, the repository of the release is never touched.

Cc Ricardo Martincoski <[email protected]>
Cc Thomas Petazzoni <[email protected]>

---
Changes v1 -> v2:
- fix ordering of packages list (Ricardo)

---
 support/docker/Dockerfile.fedora-32 | 48 +++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 support/docker/Dockerfile.fedora-32
support/docker/Dockerfile.fedora-32 new file +13 -0
new file mode 100644
index 0000000000..85e5c4ba9c
@@ -0,0 +1,48 @@
+# vi: ft=dockerfile
+# Note: this dockerfile is not reproducible, because the repository
+# where we fetch packages from is not pinned to a specific point in
+# time; i.e. we'll get whatever is current at the moment we build.
+
+FROM fedora:32
+
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+      description="Container based on Fedora 32, with everything needed to run Buildroot already installed"
+
+# Remove the repository with updates, so we only
+# get packages as they were at time of release.

support/docker/Dockerfile.fedora-32 line 23

Quoted 10 lines
support/docker/Dockerfile.fedora-32 new file +10 -0
+RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \
+    dnf -y install \
+        @development-tools \
+        bc \
+        bzip2 \
+        bzr \
+        cpio \
+        cvs \
+        file \
+        gcc-c++ \

In order to use x86 toolchains, we should install glibc.i686 package.

support/docker/Dockerfile.fedora-32

Quoted 7 lines
support/docker/Dockerfile.fedora-32 new file +6 -0
+        git \
+        mercurial \
+        ncurses-devel \
+        perl-ExtUtils-MakeMaker \
+        perl-Thread-Queue \
 
+        python-nose2 \> +        python-pexpect \

There is no such python2 package but instead

python3-nose2
python3-pexpect

support/docker/Dockerfile.fedora-32 line 34

Quoted 4 lines
support/docker/Dockerfile.fedora-32 new file +4 -0
+        qemu-system-arm \
+        qemu-system-x86 \
+        rsync \
+        subversion \

sudo ?

Quoted 15 lines
support/docker/Dockerfile.fedora-32 new file +14 -0
+        unzip \
+        wget \
+        which \
+    && dnf -y clean all
+
+RUN useradd -ms /bin/bash br-user && \
+    chown -R br-user:br-user /home/br-user && \
+    printf 'br-user ALL=NOPASSWD: /usr/bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
+
+USER br-user
+WORKDIR /home/br-user
+ENV HOME /home/br-user
+ENV LC_ALL C.utf8
#10 lore raw

v2 cover support/docker: add new dockers (branch yem/dockers)

In reply to Yann E. MORIN

Hi Yann, Thomas,

Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:

Quoted 8 lines

Hello All!

Enhance our docker support, update the existing image, add a new one.

This will help:

- users, to get a build environment known to work,

Quoted 8 lines

- developpers, to more easily reproduce build failures reported by users.

A basic set of a few, very various distros have been chosen, to cover a range of diversity:

- the existing, rock-solid Debian Stretch, pinned to a known state,

    that we use for the Gitlab-CI testing;

This Docker image could be used by toolchain-builder project instead of reinstalling every time packages needed by Buildroot:

https://github.com/bootlin/toolchains-builder/blob/master/.gitlab-ci.yml.in

Note: openssh-client is used to upload build artifacts and the toolchain archive to the Bootlin server. So we still need admin right to install some package.

Best regards,
Romain

Quoted 52 lines

- Fedora-F32, a bleeding-edge mainstream distro, pinned to the state

    it was at release time;

More variants can be added later, depending on the demand.

Note: when this series is applied (if it is), I'll generate the images and push them to the Buildroot repo on the Docker hub.

Changes v1 -> v2:
- reorder patches to ensure bisectability (Ricardo)
- drop the Makefile rule in favour of a script
- drop Arch and Alpine for now...

Regards,
Yann E. MORIN.

The following changes since commit 53158e41fdd4bcc1272bbf4e61b68f58bf9c1527

package/openocd: fix -fno-common build failure (2020-05-17 21:50:04 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 48c45cb866a160ce4cc05b03fdf15237eb346b66

support/docker: add a Fedore-32 based image (2020-05-17 23:31:36 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
support/docker: rename the Dockerfile and its accompanying file support/docker: add script to generate docker iamges support/docker: allow the br-user to becone root support/docker: add a Fedore-32 based image

 support/docker/Dockerfile.base                     |  1 +
 .../{Dockerfile => Dockerfile.debian-stretch}      | 12 ++++--
 support/docker/Dockerfile.fedora-32                | 48 ++++++++++++++++++++++
 support/docker/build-all                           | 17 ++++++++
 ...ources.list => debian-stretch.apt-sources.list} |  0
 5 files changed, 75 insertions(+), 3 deletions(-)
 create mode 120000 support/docker/Dockerfile.base
 rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (80%)
 create mode 100644 support/docker/Dockerfile.fedora-32
 create mode 100755 support/docker/build-all
 rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)