support/docker: improve and extend our docker image (branch yem/dockers)

index 5a0f298eb1..b7519cc01a 100644
@@ -44,13 +44,13 @@ project after an empty line.
 As a convention specific to Buildroot, the ordering of the attributes
 is as follows:
 
-1. The type of option: +bool+, +string+... with the prompt
-2. If needed, the +default+ value(s)
-3. Any dependencies on the target in +depends on+ form
-4. Any dependencies on the toolchain in +depends on+ form
-5. Any dependencies on other packages in +depends on+ form
-6. Any dependency of the +select+ form
-7. The help keyword and help text.
+. The type of option: +bool+, +string+... with the prompt
+. If needed, the +default+ value(s)
+. Any dependencies on the target in +depends on+ form
+. Any dependencies on the toolchain in +depends on+ form
+. Any dependencies on other packages in +depends on+ form
+. Any dependency of the +select+ form
+. The help keyword and help text.
 
 You can add other sub-options into a +if BR2_PACKAGE_LIBFOO...endif+
 statement to configure particular things in your software. You can look at

index e9c6968e79..739bed25d5 100644
@@ -23,11 +23,11 @@ Due to this, and in order to make sure that Native Language Support is
 properly handled, packages in Buildroot that can use NLS support
 should:
 
-1. Ensure NLS support is enabled when +BR2_SYSTEM_ENABLE_NLS=y+. This
+. Ensure NLS support is enabled when +BR2_SYSTEM_ENABLE_NLS=y+. This
    is done automatically for 'autotools' packages and therefore should
    only be done for packages using other package infrastructures.
 
-1. Add +$(TARGET_NLS_DEPENDENCIES)+ to the package
+. Add +$(TARGET_NLS_DEPENDENCIES)+ to the package
    +<pkg>_DEPENDENCIES+ variable. This addition should be done
    unconditionally: the value of this variable is automatically
    adjusted by the core infrastructure to contain the relevant list of

@@ -38,7 +38,7 @@ should:
    also contains +gettext+ in order to get the full-blown 'gettext'
    implementation.
 
-1. If needed, add +$(TARGET_NLS_LIBS)+ to the linker flags, so that
+. If needed, add +$(TARGET_NLS_LIBS)+ to the linker flags, so that
    the package gets linked with +libintl+. This is generally not
    needed with 'autotools' packages as they usually detect
    automatically that they should link with +libintl+. However,

index 627ecbacb1..315027c590 100644
@@ -9,11 +9,11 @@ now summarize all this by providing step-by-step instructions to storing your
 project-specific customizations. Clearly, the steps that are not relevant to
 your project can be skipped.
 
-1. +make menuconfig+ to configure toolchain, packages and kernel.
-1. +make linux-menuconfig+ to update the kernel config, similar for
+. +make menuconfig+ to configure toolchain, packages and kernel.
+. +make linux-menuconfig+ to update the kernel config, similar for
    other configuration like busybox, uclibc, ...
-1. +mkdir -p board/<manufacturer>/<boardname>+
-1. Set the following options to +board/<manufacturer>/<boardname>/<package>.config+
+. +mkdir -p board/<manufacturer>/<boardname>+
+. Set the following options to +board/<manufacturer>/<boardname>/<package>.config+
    (as far as they are relevant):
    * +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
    * +BR2_PACKAGE_BUSYBOX_CONFIG+

@@ -21,7 +21,7 @@ your project can be skipped.
    * +BR2_TARGET_AT91BOOTSTRAP3_CUSTOM_CONFIG_FILE+
    * +BR2_TARGET_BAREBOX_CUSTOM_CONFIG_FILE+
    * +BR2_TARGET_UBOOT_CUSTOM_CONFIG_FILE+
-1. Write the configuration files:
+. Write the configuration files:
    * +make linux-update-defconfig+
    * +make busybox-update-config+
    * +make uclibc-update-config+

@@ -29,38 +29,38 @@ your project can be skipped.
      board/<manufacturer>/<boardname>/at91bootstrap3.config+
    * +make barebox-update-defconfig+
    * +make uboot-update-defconfig+
-1. Create +board/<manufacturer>/<boardname>/rootfs-overlay/+ and fill it
+. Create +board/<manufacturer>/<boardname>/rootfs-overlay/+ and fill it
    with additional files you need on your rootfs, e.g.
    +board/<manufacturer>/<boardname>/rootfs-overlay/etc/inittab+.
    Set +BR2_ROOTFS_OVERLAY+
    to +board/<manufacturer>/<boardname>/rootfs-overlay+.
-1. Create a post-build script
+. Create a post-build script
    +board/<manufacturer>/<boardname>/post_build.sh+. Set
    +BR2_ROOTFS_POST_BUILD_SCRIPT+ to
    +board/<manufacturer>/<boardname>/post_build.sh+
-1. If additional setuid permissions have to be set or device nodes have
+. If additional setuid permissions have to be set or device nodes have
    to be created, create +board/<manufacturer>/<boardname>/device_table.txt+
    and add that path to +BR2_ROOTFS_DEVICE_TABLE+.
-1. If additional user accounts have to be created, create
+. If additional user accounts have to be created, create
    +board/<manufacturer>/<boardname>/users_table.txt+ and add that path
    to +BR2_ROOTFS_USERS_TABLES+.
-1. To add custom patches to certain packages, set +BR2_GLOBAL_PATCH_DIR+
+. To add custom patches to certain packages, set +BR2_GLOBAL_PATCH_DIR+
    to +board/<manufacturer>/<boardname>/patches/+ and add your patches
    for each package in a subdirectory named after the package. Each
    patch should be called +<packagename>-<num>-<description>.patch+.
-1. Specifically for the Linux kernel, there also exists the option
+. Specifically for the Linux kernel, there also exists the option
    +BR2_LINUX_KERNEL_PATCH+ with as main advantage that it can also
    download patches from a URL. If you do not need this,
    +BR2_GLOBAL_PATCH_DIR+ is preferred. U-Boot, Barebox, at91bootstrap
    and at91bootstrap3 also have separate options, but these do not
    provide any advantage over +BR2_GLOBAL_PATCH_DIR+ and will likely be
    removed in the future.
-1. If you need to add project-specific packages, create
+. If you need to add project-specific packages, create
    +package/<manufacturer>/+ and place your packages in that
    directory. Create an overall +<manufacturer>.mk+ file that
    includes the +.mk+ files of all your packages. Create an overall
    +Config.in+ file that sources the +Config.in+ files of all your
    packages. Include this +Config.in+ file from Buildroot's
    +package/Config.in+ file.
-1. +make savedefconfig+ to save the buildroot configuration.
-1. +cp defconfig configs/<boardname>_defconfig+
+. +make savedefconfig+ to save the buildroot configuration.
+. +cp defconfig configs/<boardname>_defconfig+

index 262a5153f5..6f07cba551 100644
@@ -41,6 +41,7 @@ between distributions).
 
 * Source fetching tools:
 ** +wget+
+** +git+
 
 [[requirement-optional]]
 

@@ -76,7 +77,6 @@ corresponding tool on the host system:
 +
 ** +bazaar+
 ** +cvs+
-** +git+
 ** +mercurial+
 ** +scp+
 ** +sftp+

index 577a148c1e..87931f09c1 100644
@@ -1240,7 +1240,6 @@ endif
 ifeq ($$($(2)_SITE_METHOD),svn)
 DL_TOOLS_DEPENDENCIES += svn
 else ifeq ($$($(2)_SITE_METHOD),git)
-DL_TOOLS_DEPENDENCIES += git
 ifneq ($$($(2)_GIT_LFS),)
 DL_TOOLS_DEPENDENCIES += git-lfs
 endif

@@ -1254,13 +1253,6 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
 endif # SITE_METHOD
 
-# cargo/go vendoring (may) need git
-ifeq ($$($(2)_DOWNLOAD_POST_PROCESS),cargo)
-DL_TOOLS_DEPENDENCIES += git
-else ifeq ($$($(2)_DOWNLOAD_POST_PROCESS),go)
-DL_TOOLS_DEPENDENCIES += git
-endif
-
 DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
 
 # Ensure all virtual targets are PHONY. Listed alphabetically.

index 6e7d067ccd..e18e799922 100755
@@ -154,7 +154,7 @@ fi
 
 # Check that a few mandatory programs are installed
 missing_progs="no"
-for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
+for prog in perl tar wget git cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
 	if ! which $prog > /dev/null ; then
 		echo "You must install '$prog' on your build machine";
 		missing_progs="yes"

index 8c6e2add60..dd9b9c0f93 100644
@@ -20,8 +20,8 @@ COPY apt-sources.list /etc/apt/sources.list
 
 # The container has no package lists, so need to update first
 RUN dpkg --add-architecture i386 && \
-    apt-get -o APT::Retries=3 update -y
-RUN apt-get -o APT::Retries=3 install -y --no-install-recommends \
+    apt-get -o APT::Retries=5 update -y
+RUN apt-get -o APT::Retries=5 install -y --no-install-recommends \
         bc \
         build-essential \
         bzr \

index dd9b9c0f93..b62cf32824 100644
@@ -49,6 +49,7 @@ RUN apt-get -o APT::Retries=5 install -y --no-install-recommends \
         rsync \
         shellcheck \
         subversion \
+        sudo \
         unzip \
         wget \
         && \

@@ -60,7 +61,9 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
     /usr/sbin/locale-gen
 
 RUN useradd -ms /bin/bash br-user && \
-    chown -R br-user:br-user /home/br-user
+    chown -R br-user:br-user /home/br-user && \
+    printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
 
 USER br-user
 WORKDIR /home/br-user

new file mode 120000
index 0000000000..95f9ccb9fc
@@ -0,0 +1 @@
+debian-bullseye
\ No newline at end of file

similarity index 97%
rename from support/docker/Dockerfile
rename to support/docker/debian-bullseye/Dockerfile
index b62cf32824..f45ff6b0f8 100644
@@ -1,6 +1,6 @@
 # This Dockerfile generates the docker image that gets used by Gitlab CI
 # To build it (YYYYMMDD.HHMM is the current date and time in UTC):
-#   docker build -t registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM support/docker
+#   docker build -t registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM support/docker/debian-stretch
 #   docker push registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM
 
 # We use a specific tag for the base image *and* the corresponding date

similarity index 100%
rename from support/docker/apt-sources.list
rename to support/docker/debian-bullseye/apt-sources.list

new file mode 100755
index 0000000000..d5c01771cf
@@ -0,0 +1,17 @@
+#!/bin/sh
+set -e
+
+BR2_DOCKER_VERSION="${1}"
+
+if [ -z "${BR2_DOCKER_VERSION}" ]; then
+    printf 'Usage: %s VERSION\n' "${0}"
+    exit 1
+fi
+
+BASE_DIR="${0%/*}"
+for context_dir in "${BASE_DIR}"/*/; do \
+    distro="$(basename "${context_dir}")"
+    docker build \
+        -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \
+        "${context_dir}"
+done

deleted file mode 120000
index 95f9ccb9fc..0000000000
@@ -1 +0,0 @@
-debian-bullseye
\ No newline at end of file

index d5c01771cf..6ae4ba5d25 100755
@@ -2,16 +2,31 @@
 set -e
 
 BR2_DOCKER_VERSION="${1}"
+BR2_REGISTRY="${2:-registry.gitlab.com/buildroot.org/}"
 
-if [ -z "${BR2_DOCKER_VERSION}" ]; then
-    printf 'Usage: %s VERSION\n' "${0}"
+case "${BR2_DOCKER_VERSION}" in
+  (""|*/*)
+    printf 'Usage: %s VERSION [REGSITRY]\n' "${0}"
     exit 1
-fi
+    ;;
+esac
+case "${BR2_REGISTRY}" in
+  (""|*/)   ;;
+  (*)       BR2_REGISTRY="${BR2_REGISTRY}/";;
+esac
 
 BASE_DIR="${0%/*}"
-for context_dir in "${BASE_DIR}"/*/; do \
-    distro="$(basename "${context_dir}")"
-    docker build \
-        -t "buildroot/${distro}:${BR2_DOCKER_VERSION}" \
-        "${context_dir}"
+# "ci" images use FROM on the "base" images, so need to ensure we build
+# the "base" images first (don't rely on alphabetical ordering!). Ditto,
+# the 'user' images are based on the 'ci' ones.
+for type in base ci user; do
+    for context_dir in "${BASE_DIR}"/*."${type}"/; do \
+        [ -e "${context_dir}" ] || continue
+        distro="$(basename "${context_dir}")"
+        docker build \
+            --build-arg BR2_REGISTRY="${BR2_REGISTRY}" \
+            --build-arg BR2_DOCKER_VERSION="${BR2_DOCKER_VERSION}" \
+            -t "${BR2_REGISTRY}buildroot/${distro}:${BR2_DOCKER_VERSION}" \
+            "${context_dir}"
+    done
 done

similarity index 59%
rename from support/docker/debian-bullseye/Dockerfile
rename to support/docker/debian-bullseye.base/Dockerfile
index f45ff6b0f8..a3e90a79ab 100644
@@ -1,7 +1,5 @@
-# This Dockerfile generates the docker image that gets used by Gitlab CI
-# To build it (YYYYMMDD.HHMM is the current date and time in UTC):
-#   docker build -t registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM support/docker/debian-stretch
-#   docker push registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM
+# vi: ft=dockerfile
+# Debian-based minimal image with only mandatory packages
 
 # We use a specific tag for the base image *and* the corresponding date
 # for the repository., so do not forget to update the apt-sources.list

@@ -24,32 +22,14 @@ RUN dpkg --add-architecture i386 && \
 RUN apt-get -o APT::Retries=5 install -y --no-install-recommends \
         bc \
         build-essential \
-        bzr \
         ca-certificates \
-        cmake \
         cpio \
-        cvs \
         file \
         g++-multilib \
         git \
         libc6:i386 \
-        libncurses5-dev \
         locales \
-        mercurial \
-        openssh-server \
-        python3 \
-        python3-flake8 \
-        python3-magic \
-        python3-nose2 \
-        python3-pexpect \
-        python3-pytest \
-        qemu-system-arm \
-        qemu-system-misc \
-        qemu-system-x86 \
         rsync \
-        shellcheck \
-        subversion \
-        sudo \
         unzip \
         wget \
         && \

@@ -61,9 +41,7 @@ RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
     /usr/sbin/locale-gen
 
 RUN useradd -ms /bin/bash br-user && \
-    chown -R br-user:br-user /home/br-user && \
-    printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \
-    chmod 0440 /etc/sudoers.d/br-user
+    chown -R br-user:br-user /home/br-user
 
 USER br-user
 WORKDIR /home/br-user

similarity index 100%
rename from support/docker/debian-bullseye/apt-sources.list
rename to support/docker/debian-bullseye.base/apt-sources.list

43 lines not on this page · raw message

new file mode 100644
index 0000000000..72f807b2d5
@@ -0,0 +1,32 @@
+# This Dockerfile generates the docker image that can be used by users
+# to get a workign build environment
+# To build it (YYYYMMDD.HHMM is the current date and time in UTC):
+#   docker build -t registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM support/docker/debian-stretch
+#   docker push registry.gitlab.com/buildroot.org/buildroot/base:YYYYMMDD.HHMM
+
+# Based on our reference build image
+ARG BR2_REGISTRY
+ARG BR2_DOCKER_VERSION
+FROM ${BR2_REGISTRY}buildroot/debian-bullseye.ci:${BR2_DOCKER_VERSION}
+
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+description="Container with everything needed to run Buildroot"
+
+USER root
+WORKDIR /root
+ENV HOME /root
+
+RUN apt-get -o APT::Retries=5 install -y --no-install-recommends \
+        libncurses5-dev \
+        sudo \
+        && \
+    apt-get -y autoremove && \
+    apt-get -y clean && \
+    printf 'br-user ALL=NOPASSWD: /bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
+
+USER br-user
+WORKDIR /home/br-user
+ENV HOME /home/br-user
+ENV LC_ALL en_US.UTF-8

index 6f07cba551..d15f0a972a 100644
@@ -96,3 +96,46 @@ corresponding tool on the host system:
 * Graph generation tools:
 ** +graphviz+ to use 'graph-depends' and '<pkg>-graph-depends'
 ** +python-matplotlib+ to use 'graph-build'
+
+
+=== Docker images
+
+The Buildroot project also provides a set of Docker images descriptions,
+in the directory `support/docker/`, which provide ready to use systems
+for using Buildroot. Those images are available on the
+https://gitlab.com/buildroot.org/buildroot/container_registry[docker
+registry of the Buildroot project].
+
+For a given distribution, and a given version of that distribution,
+there are up to three variants: 'base', 'ci', and 'user':
+
+* the 'base' variant is a minimalist system, which contains only the
+  minimal set of packages required for running Buildroot (i.e. the
+  packages listed as mandatory in the manual). The 'base' image shall
+  be used to verify that a package has all its dependencies defined,
+  for example by using:
+
+** `utils/test-pkg`
+
+* the 'ci' image inherits from the 'base' one, and adds the few packages
+  that are needed to run the CI jobs. The 'ci' image shall be used to
+  run the various tests and tooling in Buildroot:
+
+** `make check-package`
+** `support/testing/run-tests`
+** `utils/check-symbols`
+
+* the 'user' image inherits from the 'ci' image, and adds a few more
+  packages and setup that are useful for interactive use (e.g. the
+  `br-user` is sudoer without any password, but can just run a login
+  shell, using _exactly_ `sudo su -`).
+
+The Buildroot developers by no mean enforce or recommend using those
+images, or even suggest that Buildroot only works when using one of
+those images; instead, they are known-working environments that can be
+used where the distribution on the build machine can't be configured
+to match Buildroot's needs (e.g. on a shared server), or to provide a
+quick and easy way to test on various ditributions.
+
+Finally, those images can serve as the basis for further customisation:
+one can use them in the `FROM` directive in their own +Dockerfile+.

new file mode 100644
index 0000000000..e48a621103
@@ -0,0 +1,43 @@
+# vi: ft=dockerfile
+# Note: this dockerfile is not reproducible: even if we pin the image
+# to a specific tag and hash, the repository where we fetch packages
+# from is not pinned to a specific point in time; i.e. we'll get
+# whatever is current at the moment we build.
+
+FROM fedora:38@sha256:e9b68b3351329d72df54dfa5d10b2310e10829f6af58a8c30e0fc612d4f3b749
+
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+      description="Container based on Fedora 38, with everything needed to run Buildroot already installed"
+
+# Only keep the main repository, so we only get packages as they were
+# at the time of release, and we get no update: this helps a bit with
+# reproducibility.
+RUN rm -f /etc/yum.repos.d/fedora-*.repo && \
+    dnf -y install \
+        @development-tools \
+        bc \
+        bzip2 \
+        cpio \
+        file \
+        gcc-c++ \
+        git \
+        perl-English \
+        perl-ExtUtils-MakeMaker \
+        perl-FindBin \
+        perl-IPC-Cmd \
+        perl-Thread-Queue \
+        rsync \
+        unzip \
+        wget \
+        which \
+        && \
+    dnf -y clean all
+
+RUN useradd -ms /bin/bash br-user && \
+    chown -R br-user:br-user /home/br-user
+
+USER br-user
+WORKDIR /home/br-user
+ENV HOME /home/br-user
+ENV LC_ALL C.utf8

47 lines not on this page · raw message

new file mode 100644
index 0000000000..fd0ebccade
@@ -0,0 +1,35 @@
+# vi: ft=dockerfile
+# Note: this dockerfile is not reproducible: even if we pin the image
+# to a specific tag and hash, the repository where we fetch packages
+# from is not pinned to a specific point in time; i.e. we'll get
+# whatever is current at the moment we build.
+
+# Based on our reference build image
+ARG BR2_REGISTRY
+ARG BR2_DOCKER_VERSION
+FROM ${BR2_REGISTRY}buildroot/fedora-38.ci:${BR2_DOCKER_VERSION}
+
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+      description="Container based on Fedora 38, with everything needed to run Buildroot already installed"
+
+USER root
+WORKDIR /root
+ENV HOME /root
+
+# Only keep the main repository, so we only get packages as they were
+# at the time of release, and we get no update: this helps a bit with
+# reproducibility.
+RUN rm -f /etc/yum.repos.d/fedora-*.repo && \
+    dnf -y install \
+        ncurses-devel \
+        sudo \
+        && \
+    dnf -y clean all && \
+    printf 'br-user ALL=NOPASSWD: /usr/bin/su -\n' >/etc/sudoers.d/br-user && \
+    chmod 0440 /etc/sudoers.d/br-user
+
+USER br-user
+WORKDIR /home/br-user
+ENV HOME /home/br-user
+ENV LC_ALL C.utf8