support/docker: add new dockers (branch yem/dockers)
v2 cover support/docker: add new dockers (branch yem/dockers)
Hello All!
Enhance our docker support, update the existing image, add a new one.
This will help:
- users, to get a build environment known to work,
- developpers, to more easily reproduce build failures reported by users.
A basic set of a few, very various distros have been chosen, to cover a range of diversity:
- the existing, rock-solid Debian Stretch, pinned to a known state,
that we use for the Gitlab-CI testing;
- Fedora-F32, a bleeding-edge mainstream distro, pinned to the state
it was at release time;
More variants can be added later, depending on the demand.
Note: when this series is applied (if it is), I'll generate the images and push them to the Buildroot repo on the Docker hub.
Changes v1 -> v2:
- reorder patches to ensure bisectability (Ricardo)
- drop the Makefile rule in favour of a script
- drop Arch and Alpine for now...
Regards,
Yann E. MORIN.
The following changes since commit 53158e41fdd4bcc1272bbf4e61b68f58bf9c1527
package/openocd: fix -fno-common build failure (2020-05-17 21:50:04 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to 48c45cb866a160ce4cc05b03fdf15237eb346b66
support/docker: add a Fedore-32 based image (2020-05-17 23:31:36 +0200)
----------------------------------------------------------------
Yann E. MORIN (4):
support/docker: rename the Dockerfile and its accompanying file support/docker: add script to generate docker iamges support/docker: allow the br-user to becone root support/docker: add a Fedore-32 based image
support/docker/Dockerfile.base | 1 +
.../{Dockerfile => Dockerfile.debian-stretch} | 12 ++++--
support/docker/Dockerfile.fedora-32 | 48 ++++++++++++++++++++++
support/docker/build-all | 17 ++++++++
...ources.list => debian-stretch.apt-sources.list} | 0
5 files changed, 75 insertions(+), 3 deletions(-)
create mode 120000 support/docker/Dockerfile.base
rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (80%)
create mode 100644 support/docker/Dockerfile.fedora-32
create mode 100755 support/docker/build-all
rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/4 support/docker: rename the Dockerfile and its accompanying file
Currently, we only generate one docker image, for use by the gitlab-ci infra.
However, users have expressed an interest in using that image for other uses, like:
- reproducing build failures as reported by the gitlab-ci jobs;
- having a ready-to-use base image to use as-is, or as a template for further customisations.
Additionally, some users have reported the need for similar base images, but based off different distros, with the same goals as above, plus:
- reproducing build failures reported by users on other distros.
As a first step toward providing such images, rename our Dockerfile to include the name and version of the distro it is based off.
We add a symlink so that we still have a 'base' image that we can reference from the gitlab-ci infra.
---
support/docker/Dockerfile.base | 1 +
support/docker/{Dockerfile => Dockerfile.debian-stretch} | 7 +++++--
.../{apt-sources.list => debian-stretch.apt-sources.list} | 0
3 files changed, 6 insertions(+), 2 deletions(-)
create mode 120000 support/docker/Dockerfile.base
rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (88%)
rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)support/docker/Dockerfile.base new file +1 -0
@@ -0,0 +1 @@ +Dockerfile.debian-stretch
support/docker/Dockerfile.debian-stretch renamed from support/docker/Dockerfile +5 -2
@@ -1,6 +1,9 @@ +# vi: ft=dockerfile # This Dockerfile generates the docker image that gets used by Gitlab CI # To build it (YYYYMMDD.HHMM is the current date and time in UTC): -# sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker +# sudo docker build -t buildroot/base:YYYYMMDD.HHMM \ +# -f support/docker/Dockerfile.debian-stretch \ +# support/docker # sudo docker push buildroot/base:YYYYMMDD.HHMM # We use a specific tag for the base image *and* the corresponding date
@@ -16,7 +19,7 @@ description="Container with everything needed to run Buildroot" ENV DEBIAN_FRONTEND noninteractive # This repository can be a bit slow at times. Don't panic... -COPY apt-sources.list /etc/apt/sources.list +COPY debian-stretch.apt-sources.list /etc/apt/sources.list # The container has no package lists, so need to update first RUN dpkg --add-architecture i386 && \
support/docker/debian-stretch.apt-sources.list renamed from support/docker/apt-sources.list +0 -0
Signature · 1 line
2.20.1
PATCH v2 1/4 support/docker: rename the Dockerfile and its accompanying file
Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:
Quoted 19 lines
Currently, we only generate one docker image, for use by the gitlab-ci infra.
However, users have expressed an interest in using that image for other uses, like:
- reproducing build failures as reported by the gitlab-ci jobs;
- having a ready-to-use base image to use as-is, or as a template for further customisations.
Additionally, some users have reported the need for similar base images, but based off different distros, with the same goals as above, plus:
- reproducing build failures reported by users on other distros.
As a first step toward providing such images, rename our Dockerfile to include the name and version of the distro it is based off.
Quoted 6 lines
We add a symlink so that we still have a 'base' image that we can reference from the gitlab-ci infra.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Ricardo Martincoski <[email protected]>Cc Thomas Petazzoni <[email protected]>
Best regards,
Romain
Quoted 49 lines
--- support/docker/Dockerfile.base | 1 + support/docker/{Dockerfile => Dockerfile.debian-stretch} | 7 +++++-- .../{apt-sources.list => debian-stretch.apt-sources.list} | 0 3 files changed, 6 insertions(+), 2 deletions(-) create mode 120000 support/docker/Dockerfile.base rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (88%) rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)support/docker/Dockerfile.base new file +1
-0@@ -0,0 +1 @@ +Dockerfile.debian-stretchsupport/docker/Dockerfile.debian-stretch renamed from support/docker/Dockerfile +5
-2@@ -1,6 +1,9 @@ +# vi: ft=dockerfile # This Dockerfile generates the docker image that gets used by Gitlab CI # To build it (YYYYMMDD.HHMM is the current date and time in UTC): -# sudo docker build -t buildroot/base:YYYYMMDD.HHMM support/docker +# sudo docker build -t buildroot/base:YYYYMMDD.HHMM \ +# -f support/docker/Dockerfile.debian-stretch \ +# support/docker # sudo docker push buildroot/base:YYYYMMDD.HHMM # We use a specific tag for the base image *and* the corresponding date@@ -16,7 +19,7 @@ description="Container with everything needed to run Buildroot" ENV DEBIAN_FRONTEND noninteractive # This repository can be a bit slow at times. Don't panic... -COPY apt-sources.list /etc/apt/sources.list +COPY debian-stretch.apt-sources.list /etc/apt/sources.list # The container has no package lists, so need to update first RUN dpkg --add-architecture i386 && \support/docker/debian-stretch.apt-sources.list renamed from support/docker/apt-sources.list +0
-0
PATCH v2 2/4 support/docker: add script to generate docker iamges
In reply to Yann E. MORIN--- support/docker/build-all | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 support/docker/build-all
support/docker/build-all new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh +set -e + +BR2_DOCKER_VERSION="${1}" + +if [ -z "${BR2_DOCKER_VERSION}" ]; then + printf "Usage: ${0} VERSION\n" + exit 1 +fi + +CONTEXT_DIR="${0%/*}" +for d in "${CONTEXT_DIR}"/Dockerfile.*; do \ + distro="${d#*/Dockerfile.}" + docker build \ + -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \ + -f "${d}" "${CONTEXT_DIR}" +done
Signature · 1 line
2.20.1
PATCH v2 2/4 support/docker: add script to generate docker iamges
Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:
Quoted 14 lines
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Ricardo Martincoski <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/docker/build-all | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 support/docker/build-allsupport/docker/build-all new file +0
-0
support/docker/build-all line 2
Quoted 2 lines
support/docker/build-all new file +2
-0@@ -0,0 +1,17 @@ +#!/bin/sh +set -e
Best regards,
Romain
Quoted 16 lines
support/docker/build-all new file +15
-0+ +BR2_DOCKER_VERSION="${1}" + +if [ -z "${BR2_DOCKER_VERSION}" ]; then + printf "Usage: ${0} VERSION\n" + exit 1 +fi + +CONTEXT_DIR="${0%/*}" +for d in "${CONTEXT_DIR}"/Dockerfile.*; do \ + distro="${d#*/Dockerfile.}" + docker build \ + -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \ + -f "${d}" "${CONTEXT_DIR}" +done
PATCH v2 3/4 support/docker: allow the br-user to becone root
In reply to Yann E. MORINWhen our Dockerfiles (and images) get reused by users, they might need to locally add new packages or tweak the iamge to heir local needs.
As such, register the br-user to become root by running (exactly!):
$ sudo su -
This will allow acutally becoing root without prompting for a password, but no other command will be accepted, and no other paramter will be accepted either.
Cc; Thomas Petazzoni <[email protected]>
--- support/docker/Dockerfile.debian-stretch | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
support/docker/Dockerfile.debian-stretch +4 -1
@@ -49,6 +49,7 @@ RUN apt-get install -y --no-install-recommends \ qemu-system-x86 \ rsync \ subversion \ + sudo \ unzip \ wget \ && \
@@ -60,7 +61,9 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \ /usr/sbin/locale-gen RUN useradd -ms /bin/bash br-user && \ - chown -R br-user:br-user /home/br-user + chown -R br-user:br-user /home/br-user && \ + printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \ + chmod 0440 /etc/sudoers.d/br-user USER br-user WORKDIR /home/br-user
Signature · 1 line
2.20.1
PATCH v2 3/4 support/docker: allow the br-user to becone root
Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:
Quoted 4 lines
When our Dockerfiles (and images) get reused by users, they might need to locally add new packages or tweak the iamge to heir local needs.
As such, register the br-user to become root by running (exactly!):
Quoted 8 lines
$ sudo su -This will allow acutally becoing root without prompting for a password, but no other command will be accepted, and no other paramter will be accepted either.
Signed-off-by Yann E. MORIN <[email protected]>Cc; Thomas Petazzoni <[email protected]>
Best regards,
Romain
Quoted 28 lines
--- support/docker/Dockerfile.debian-stretch | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)support/docker/Dockerfile.debian-stretch +4
-1@@ -49,6 +49,7 @@ RUN apt-get install -y --no-install-recommends \ qemu-system-x86 \ rsync \ subversion \ + sudo \ unzip \ wget \ && \@@ -60,7 +61,9 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \ /usr/sbin/locale-gen RUN useradd -ms /bin/bash br-user && \ - chown -R br-user:br-user /home/br-user + chown -R br-user:br-user /home/br-user && \ + printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \ + chmod 0440 /etc/sudoers.d/br-user USER br-user WORKDIR /home/br-user
PATCH v2 4/4 support/docker: add a Fedore-32 based image
In reply to Yann E. MORINHopefully, this image is reproducible, since we are not using any update repository, and from the looks of it, the repository of the release is never touched.
---
Changes v1 -> v2:
- fix ordering of packages list (Ricardo)
--- support/docker/Dockerfile.fedora-32 | 48 +++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 support/docker/Dockerfile.fedora-32
support/docker/Dockerfile.fedora-32 new file +48 -0
@@ -0,0 +1,48 @@ +# vi: ft=dockerfile +# Note: this dockerfile is not reproducible, because the repository +# where we fetch packages from is not pinned to a specific point in +# time; i.e. we'll get whatever is current at the moment we build. + +FROM fedora:32 + +LABEL maintainer="Buildroot mailing list <[email protected]>" \ + vendor="Buildroot" \ + description="Container based on Fedora 32, with everything needed to run Buildroot already installed" + +# Remove the repository with updates, so we only +# get packages as they were at time of release. +RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \ + dnf -y install \ + @development-tools \ + bc \ + bzip2 \ + bzr \ + cpio \ + cvs \ + file \ + gcc-c++ \ + git \ + mercurial \ + ncurses-devel \ + perl-ExtUtils-MakeMaker \ + perl-Thread-Queue \ + python-nose2 \ + python-pexpect \ + qemu-system-arm \ + qemu-system-x86 \ + rsync \ + subversion \ + unzip \ + wget \ + which \ + && dnf -y clean all + +RUN useradd -ms /bin/bash br-user && \ + chown -R br-user:br-user /home/br-user && \ + printf 'br-user ALL=NOPASSWD: /usr/bin/su -\n' >/etc/sudoers.d/br-user && \ + chmod 0440 /etc/sudoers.d/br-user + +USER br-user +WORKDIR /home/br-user +ENV HOME /home/br-user +ENV LC_ALL C.utf8
Signature · 1 line
2.20.1
PATCH v2 4/4 support/docker: add a Fedore-32 based image
Hello,
Thanks for doing this work, I did need some Docker images just yesterday. One comment/question below.
Quoted 11 lines · On Sun, 17 May 2020 23:41:40 +0200 "Yann E. MORIN" <[email protected]>…
support/docker/Dockerfile.fedora-32 new file +11
-0+# Remove the repository with updates, so we only +# get packages as they were at time of release. +RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \ + dnf -y install \ + @development-tools \ + bc \ + bzip2 \ + bzr \ + cpio \ + cvs \ + file \
support/docker/Dockerfile.fedora-32 line 32
Quoted 10 lines
support/docker/Dockerfile.fedora-32 new file +10
-0+ gcc-c++ \ + git \ + mercurial \ + ncurses-devel \ + perl-ExtUtils-MakeMaker \ + perl-Thread-Queue \ + python-nose2 \ + python-pexpect \ + qemu-system-arm \ + qemu-system-x86 \
With python-nose2 and python-pexpect, this pulls in a Python interpreter, which is not strictly a requirement of Buildroot. I know these are needed to run the runtime tests, and therefore to run the Gitlab CI stuff. But sometimes we need to do some tests with really the minimal set of Buildroot dependencies. Same goes with qemu-system-* packages.
Should we have two sets of Docker images, one with the minimal set of dependencies, and one that adds the stuff needed for Gitlab CI ? I believe Docker allows to create some kind of layered "thing" where an image sits on top of another.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 4/4 support/docker: add a Fedore-32 based image
In reply to Yann E. MORINCommit tittle : Fedore/Fedora
Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:
Quoted 35 lines
Hopefully, this image is reproducible, since we are not using any update repository, and from the looks of it, the repository of the release is never touched.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Ricardo Martincoski <[email protected]>Cc Thomas Petazzoni <[email protected]>---
Changes v1 -> v2:
- fix ordering of packages list (Ricardo)--- support/docker/Dockerfile.fedora-32 | 48 +++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 support/docker/Dockerfile.fedora-32support/docker/Dockerfile.fedora-32 new file +13
-0@@ -0,0 +1,48 @@ +# vi: ft=dockerfile +# Note: this dockerfile is not reproducible, because the repository +# where we fetch packages from is not pinned to a specific point in +# time; i.e. we'll get whatever is current at the moment we build. + +FROM fedora:32 + +LABEL maintainer="Buildroot mailing list <[email protected]>" \ + vendor="Buildroot" \ + description="Container based on Fedora 32, with everything needed to run Buildroot already installed" + +# Remove the repository with updates, so we only +# get packages as they were at time of release.
support/docker/Dockerfile.fedora-32 line 23
Quoted 10 lines
support/docker/Dockerfile.fedora-32 new file +10
-0+RUN rm -f /etc/yum.repos.d/fedora-updates*.repo && \ + dnf -y install \ + @development-tools \ + bc \ + bzip2 \ + bzr \ + cpio \ + cvs \ + file \ + gcc-c++ \
In order to use x86 toolchains, we should install glibc.i686 package.
support/docker/Dockerfile.fedora-32
Quoted 7 lines
support/docker/Dockerfile.fedora-32 new file +6
-0+ git \ + mercurial \ + ncurses-devel \ + perl-ExtUtils-MakeMaker \ + perl-Thread-Queue \ + python-nose2 \> + python-pexpect \
There is no such python2 package but instead
python3-nose2
python3-pexpect
support/docker/Dockerfile.fedora-32 line 34
Quoted 4 lines
support/docker/Dockerfile.fedora-32 new file +4
-0+ qemu-system-arm \ + qemu-system-x86 \ + rsync \ + subversion \
sudo ?
Quoted 15 lines
support/docker/Dockerfile.fedora-32 new file +14
-0+ unzip \ + wget \ + which \ + && dnf -y clean all + +RUN useradd -ms /bin/bash br-user && \ + chown -R br-user:br-user /home/br-user && \ + printf 'br-user ALL=NOPASSWD: /usr/bin/su -\n' >/etc/sudoers.d/br-user && \ + chmod 0440 /etc/sudoers.d/br-user + +USER br-user +WORKDIR /home/br-user +ENV HOME /home/br-user +ENV LC_ALL C.utf8
v2 cover support/docker: add new dockers (branch yem/dockers)
In reply to Yann E. MORINHi Yann, Thomas,
Le 17/05/2020 ? 23:41, Yann E. MORIN a ?crit?:
Quoted 8 lines
Hello All!
Enhance our docker support, update the existing image, add a new one.
This will help:
- users, to get a build environment known to work,
Quoted 8 lines
- developpers, to more easily reproduce build failures reported by users.
A basic set of a few, very various distros have been chosen, to cover a range of diversity:
- the existing, rock-solid Debian Stretch, pinned to a known state,
that we use for the Gitlab-CI testing;
This Docker image could be used by toolchain-builder project instead of reinstalling every time packages needed by Buildroot:
https://github.com/bootlin/toolchains-builder/blob/master/.gitlab-ci.yml.in
Note: openssh-client is used to upload build artifacts and the toolchain archive to the Bootlin server. So we still need admin right to install some package.
Best regards,
Romain
Quoted 52 lines
- Fedora-F32, a bleeding-edge mainstream distro, pinned to the state
it was at release time;More variants can be added later, depending on the demand.
Note: when this series is applied (if it is), I'll generate the images and push them to the Buildroot repo on the Docker hub.
Changes v1 -> v2:
- reorder patches to ensure bisectability (Ricardo)
- drop theMakefilerule in favour of a script
- drop Arch and Alpine for now...Regards,
Yann E. MORIN.The following changes since commit
53158e41fdd4bcc1272bbf4e61b68f58bf9c1527package/openocd: fix -fno-common build failure (2020-05-17 21:50:04 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.gitfor you to fetch changes up to
48c45cb866a160ce4cc05b03fdf15237eb346b66support/docker: add a Fedore-32 based image (2020-05-17 23:31:36 +0200)
----------------------------------------------------------------
Yann E. MORIN (4):
support/docker: rename the Dockerfile and its accompanying file support/docker: add script to generate docker iamges support/docker: allow the br-user to becone root support/docker: add a Fedore-32 based imagesupport/docker/Dockerfile.base | 1 + .../{Dockerfile => Dockerfile.debian-stretch} | 12 ++++-- support/docker/Dockerfile.fedora-32 | 48 ++++++++++++++++++++++ support/docker/build-all | 17 ++++++++ ...ources.list => debian-stretch.apt-sources.list} | 0 5 files changed, 75 insertions(+), 3 deletions(-) create mode 120000 support/docker/Dockerfile.base rename support/docker/{Dockerfile => Dockerfile.debian-stretch} (80%) create mode 100644 support/docker/Dockerfile.fedora-32 create mode 100755 support/docker/build-all rename support/docker/{apt-sources.list => debian-stretch.apt-sources.list} (100%)