Skip to contentSkip to conversation

docker: misc cleanups

PATCH In review John Snow 3 participants 16 messages Sep 20, 2019 – Sep 23, 2019

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover docker: misc cleanups

Another handful of misc tidyups.

Note: this WILL conflict with the other cleanups already in-flight, but I didn't have a convenient branch to base against at the time; but it should be pretty trivial to resolve/respin.

John Snow (3):
docker: remove 'deprecated' image definitions
docker: remove debian8-mxe definitions
docker: remove unused debian-sid and debian-ports

 tests/docker/Makefile.include                |  9 +++--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 4 insertions(+), 76 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
Signature · 1 line
2.21.0
author
#2 lore raw

PATCH 2/3 docker: remove debian8-mxe definitions

We don't have a debian8-mxe dockerfile anymore.

---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/docker/Makefile.include +1 -1
index 8825821127..7af476d957 100644
@@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
 DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
-DEBIAN_PARTIAL_IMAGES += debian8-mxe debian-9-mxe debian-ports debian-bootstrap
+DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-ports debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/3 docker: remove 'deprecated' image definitions

In reply to John Snow

There isn't a debian.dockerfile anymore,
so perform some ghost-busting.

---
 tests/docker/Makefile.include | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tests/docker/Makefile.include +3 -4
index 50a400b573..8825821127 100644
@@ -4,11 +4,10 @@
 
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
-DOCKER_DEPRECATED_IMAGES := debian
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian10 debian-sid
+DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
 DEBIAN_PARTIAL_IMAGES += debian8-mxe debian-9-mxe debian-ports debian-bootstrap
-DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
+DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
 DOCKER_CCACHE_DIR := $$HOME/.cache/qemu-docker-ccache
@@ -158,7 +157,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
+$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
Signature · 1 line
2.21.0
author
#3 lore raw

PATCH 3/3 docker: remove unused debian-sid and debian-ports

In reply to John Snow

These are listed as "partial" images, but have no user. Remove them.

---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
tests/docker/Makefile.include +2 -2
index 7af476d957..c282b5c520 100644
@@ -5,8 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
-DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-ports debian-bootstrap
+DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10
+DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
tests/docker/dockerfiles/debian-ports.docker deleted +0 -36
deleted file mode 100644
index 61bc3f2993..0000000000
@@ -1,36 +0,0 @@
-#
-# Docker multiarch cross-compiler target
-#
-# This docker target is builds on Debian Ports cross compiler targets
-# to build distro with a selection of cross compilers for building test binaries.
-#
-# On its own you can't build much but the docker-foo-cross targets
-# build on top of the base debian image.
-#
-FROM debian:unstable
-
-MAINTAINER Philippe Mathieu-Daudé <[email protected]>
-
-RUN echo "deb [arch=amd64] http://deb.debian.org/debian unstable main" > /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed -ne "s/^deb\ \(\[.*\]\ \)\?\(.*\)/deb-src \2/p" >> /etc/apt/sources.list
-
-# Setup some basic tools we need
-RUN apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt-get install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        clang \
-        debian-ports-archive-keyring \
-        flex \
-        gettext \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo \
-        $(apt-get -s build-dep qemu | egrep ^Inst | fgrep '[all]' | cut -d\  -f2)
tests/docker/dockerfiles/debian-sid.docker deleted +0 -35
deleted file mode 100644
index 2a1bcc33b2..0000000000
@@ -1,35 +0,0 @@
-#
-# Debian Sid Base
-#
-# Currently we can build all our guests with cross-compilers in the
-# latest Debian release (Buster). However new compilers will first
-# arrive in Sid. However Sid is a rolling distro which may be broken
-# at any particular time. To try and mitigate this we use Debian's
-# snapshot archive which provides a "stable" view of what state Sid
-# was in.
-#
-
-# This must be earlier than the snapshot date we are aiming for
-FROM debian:sid-20190812-slim
-
- # Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
-ENV DEBIAN_SNAPSHOT_DATE "20190820"
-RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        flex \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
Signature · 1 line
2.21.0
#4 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

Quoted 108 lines
---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
tests/docker/Makefile.include +2 -2
index 7af476d957..c282b5c520 100644
@@ -5,8 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
-DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-ports debian-bootstrap
+DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10
+DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
tests/docker/dockerfiles/debian-ports.docker deleted +0 -36
deleted file mode 100644
index 61bc3f2993..0000000000
@@ -1,36 +0,0 @@
-#
-# Docker multiarch cross-compiler target
-#
-# This docker target is builds on Debian Ports cross compiler targets
-# to build distro with a selection of cross compilers for building test binaries.
-#
-# On its own you can't build much but the docker-foo-cross targets
-# build on top of the base debian image.
-#
-FROM debian:unstable
-
-MAINTAINER Philippe Mathieu-Daudé <[email protected]>
-
-RUN echo "deb [arch=amd64] http://deb.debian.org/debian unstable main" > /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed -ne "s/^deb\ \(\[.*\]\ \)\?\(.*\)/deb-src \2/p" >> /etc/apt/sources.list
-
-# Setup some basic tools we need
-RUN apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt-get install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        clang \
-        debian-ports-archive-keyring \
-        flex \
-        gettext \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo \
-        $(apt-get -s build-dep qemu | egrep ^Inst | fgrep '[all]' | cut -d\  -f2)
tests/docker/dockerfiles/debian-sid.docker deleted +0 -35
deleted file mode 100644
index 2a1bcc33b2..0000000000
@@ -1,35 +0,0 @@
-#
-# Debian Sid Base
-#
-# Currently we can build all our guests with cross-compilers in the
-# latest Debian release (Buster). However new compilers will first
-# arrive in Sid. However Sid is a rolling distro which may be broken
-# at any particular time. To try and mitigate this we use Debian's
-# snapshot archive which provides a "stable" view of what state Sid
-# was in.
-#
-
-# This must be earlier than the snapshot date we are aiming for
-FROM debian:sid-20190812-slim
-
- # Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
-ENV DEBIAN_SNAPSHOT_DATE "20190820"
-RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        flex \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
#6 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to John Snow
Quoted 17 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

Quoted 8 lines

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

Quoted 105 lines

You can always re-add them when you have a user.

--js

Quoted 100 lines
Quoted 100 lines
---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
tests/docker/Makefile.include +2 -2
index 7af476d957..c282b5c520 100644
@@ -5,8 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
-DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-ports debian-bootstrap
+DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10
+DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
tests/docker/dockerfiles/debian-ports.docker deleted +0 -36
deleted file mode 100644
index 61bc3f2993..0000000000
@@ -1,36 +0,0 @@
-#
-# Docker multiarch cross-compiler target
-#
-# This docker target is builds on Debian Ports cross compiler targets
-# to build distro with a selection of cross compilers for building test binaries.
-#
-# On its own you can't build much but the docker-foo-cross targets
-# build on top of the base debian image.
-#
-FROM debian:unstable
-
-MAINTAINER Philippe Mathieu-Daudé <[email protected]>
-
-RUN echo "deb [arch=amd64] http://deb.debian.org/debian unstable main" > /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed -ne "s/^deb\ \(\[.*\]\ \)\?\(.*\)/deb-src \2/p" >> /etc/apt/sources.list
-
-# Setup some basic tools we need
-RUN apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt-get install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        clang \
-        debian-ports-archive-keyring \
-        flex \
-        gettext \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo \
-        $(apt-get -s build-dep qemu | egrep ^Inst | fgrep '[all]' | cut -d\  -f2)
tests/docker/dockerfiles/debian-sid.docker deleted +0 -28
deleted file mode 100644
index 2a1bcc33b2..0000000000
@@ -1,35 +0,0 @@
-#
-# Debian Sid Base
-#
-# Currently we can build all our guests with cross-compilers in the
-# latest Debian release (Buster). However new compilers will first
-# arrive in Sid. However Sid is a rolling distro which may be broken
-# at any particular time. To try and mitigate this we use Debian's
-# snapshot archive which provides a "stable" view of what state Sid
-# was in.
-#
-
-# This must be earlier than the snapshot date we are aiming for
-FROM debian:sid-20190812-slim
-
- # Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
-ENV DEBIAN_SNAPSHOT_DATE "20190820"
-RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        bison \
-        build-essential \
Quoted 8 lines
Quoted 8 lines

tests/docker/dockerfiles/debian-sid.docker line 35

Quoted 8 lines
tests/docker/dockerfiles/debian-sid.docker deleted +0 -7
-        ca-certificates \
-        flex \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }

--
Alex Bennée

#8 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Philippe Mathieu-Daudé
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Alex Bennée
Quoted 25 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Quoted 5 lines
Quoted 3 lines

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

For Sid I have this case:

Does it really need sid to build binutils/gcc?

Quoted 45 lines

-- >8 --
#
# Docker Renesas RX cross-compiler target
#
# This docker target builds on the debian Sid base image. #
# Copyright (c) 2019 Philippe Mathieu-Daudé
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
FROM qemu:debian-sid

MAINTAINER Philippe Mathieu-Daudé <[email protected]>

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
    apt install -y --no-install-recommends \
        wget
RUN wget -qO - https://ftpmirror.gnu.org/binutils/binutils-2.32.tar.xz \
    | tar -C /usr/src -xJf - && \
    cd /usr/src/binutils-2.32 && \
        CFLAGS=-w \
        ./configure --prefix=/usr --disable-nls
--target=rx-unknown-linux && \
        make && make install && \
        rm -rf /usr/src/binutils-2.32

RUN git clone -b rx-trunk http://pf.osdn.net/gitroot/y/ys/ysato/gcc.git \
/usr/src/gcc

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
apt install -y --no-install-recommends \
libgmp-dev libmpfr-dev libmpc-dev

RUN cd /usr/src/gcc && \
    CPPFLAGS=-w \
    ./configure --target=rx-unknown-linux \
        --prefix=/usr --disable-nls \
        --enable-languages=c --disable-shared \
        --disable-threads --with-uclibc \
        --disable-libssp --disable-libquadmath \
        --disable-libgomp --disable-libatomic && \
    make && make install && \
    rm -rf /usr/src/gcc
Quoted 3 lines

# This image isn't designed for building QEMU but building tests ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
---

--
Alex Bennée

#7 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Alex Bennée
Quoted 23 lines · Alex Bennée
Quoted 22 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

Quoted 7 lines
Quoted 3 lines

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

For Sid I have this case:

-- >8 --
#
# Docker Renesas RX cross-compiler target
#
# This docker target builds on the debian Sid base image. #
# Copyright (c) 2019 Philippe Mathieu-Daudé
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
FROM qemu:debian-sid

MAINTAINER Philippe Mathieu-Daudé <[email protected]>

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
    apt install -y --no-install-recommends \
        wget
RUN wget -qO - https://ftpmirror.gnu.org/binutils/binutils-2.32.tar.xz \
    | tar -C /usr/src -xJf - && \
    cd /usr/src/binutils-2.32 && \
        CFLAGS=-w \
        ./configure --prefix=/usr --disable-nls
--target=rx-unknown-linux && \
        make && make install && \
        rm -rf /usr/src/binutils-2.32

RUN git clone -b rx-trunk http://pf.osdn.net/gitroot/y/ys/ysato/gcc.git \
/usr/src/gcc

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
apt install -y --no-install-recommends \
libgmp-dev libmpfr-dev libmpc-dev

RUN cd /usr/src/gcc && \
    CPPFLAGS=-w \
    ./configure --target=rx-unknown-linux \
        --prefix=/usr --disable-nls \
        --enable-languages=c --disable-shared \
        --disable-threads --with-uclibc \
        --disable-libssp --disable-libquadmath \
        --disable-libgomp --disable-libatomic && \
    make && make install && \
    rm -rf /usr/src/gcc

# This image isn't designed for building QEMU but building tests ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
---

author
#5 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Philippe Mathieu-Daudé
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

You can always re-add them when you have a user.

--js

Quoted 108 lines
Quoted 108 lines
---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
tests/docker/Makefile.include +2 -2
index 7af476d957..c282b5c520 100644
@@ -5,8 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10 debian-sid
-DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-ports debian-bootstrap
+DOCKER_PARTIAL_IMAGES := debian8 debian9 debian10
+DEBIAN_PARTIAL_IMAGES += debian-9-mxe debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
tests/docker/dockerfiles/debian-ports.docker deleted +0 -36
deleted file mode 100644
index 61bc3f2993..0000000000
@@ -1,36 +0,0 @@
-#
-# Docker multiarch cross-compiler target
-#
-# This docker target is builds on Debian Ports cross compiler targets
-# to build distro with a selection of cross compilers for building test binaries.
-#
-# On its own you can't build much but the docker-foo-cross targets
-# build on top of the base debian image.
-#
-FROM debian:unstable
-
-MAINTAINER Philippe Mathieu-Daudé <[email protected]>
-
-RUN echo "deb [arch=amd64] http://deb.debian.org/debian unstable main" > /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed -ne "s/^deb\ \(\[.*\]\ \)\?\(.*\)/deb-src \2/p" >> /etc/apt/sources.list
-
-# Setup some basic tools we need
-RUN apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt-get install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        clang \
-        debian-ports-archive-keyring \
-        flex \
-        gettext \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo \
-        $(apt-get -s build-dep qemu | egrep ^Inst | fgrep '[all]' | cut -d\  -f2)
tests/docker/dockerfiles/debian-sid.docker deleted +0 -35
deleted file mode 100644
index 2a1bcc33b2..0000000000
@@ -1,35 +0,0 @@
-#
-# Debian Sid Base
-#
-# Currently we can build all our guests with cross-compilers in the
-# latest Debian release (Buster). However new compilers will first
-# arrive in Sid. However Sid is a rolling distro which may be broken
-# at any particular time. To try and mitigate this we use Debian's
-# snapshot archive which provides a "stable" view of what state Sid
-# was in.
-#
-
-# This must be earlier than the snapshot date we are aiming for
-FROM debian:sid-20190812-slim
-
- # Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
-ENV DEBIAN_SNAPSHOT_DATE "20190820"
-RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        bison \
-        build-essential \
-        ca-certificates \
-        flex \
-        git \
-        pkg-config \
-        psmisc \
-        python \
-        texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
#9 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Philippe Mathieu-Daudé
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 23 lines · Alex Bennée
Quoted 22 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

Quoted 7 lines
Quoted 3 lines

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

For Sid I have this case:

Quoted 11 lines

-- >8 --
#
# Docker Renesas RX cross-compiler target
#
# This docker target builds on the debian Sid base image. #
# Copyright (c) 2019 Philippe Mathieu-Daudé
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
FROM qemu:debian-sid

Oh well nevermind it is old, so it probably now works with debian-10.

I used the Ports base for sparc32 and x32, I don't have them on my workstation I'll check during the WE.

Quoted 36 lines

MAINTAINER Philippe Mathieu-Daudé <[email protected]>

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
    apt install -y --no-install-recommends \
        wget
RUN wget -qO - https://ftpmirror.gnu.org/binutils/binutils-2.32.tar.xz \
    | tar -C /usr/src -xJf - && \
    cd /usr/src/binutils-2.32 && \
        CFLAGS=-w \
        ./configure --prefix=/usr --disable-nls
--target=rx-unknown-linux && \
        make && make install && \
        rm -rf /usr/src/binutils-2.32

RUN git clone -b rx-trunk http://pf.osdn.net/gitroot/y/ys/ysato/gcc.git \
/usr/src/gcc

RUN DEBIAN_FRONTEND=noninteractive eatmydata \
apt install -y --no-install-recommends \
libgmp-dev libmpfr-dev libmpc-dev

RUN cd /usr/src/gcc && \
    CPPFLAGS=-w \
    ./configure --target=rx-unknown-linux \
        --prefix=/usr --disable-nls \
        --enable-languages=c --disable-shared \
        --disable-threads --with-uclibc \
        --disable-libssp --disable-libquadmath \
        --disable-libgomp --disable-libatomic && \
    make && make install && \
    rm -rf /usr/src/gcc

# This image isn't designed for building QEMU but building tests ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
---

author
#11 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Alex Bennée
Quoted 25 lines · Alex Bennée
Quoted 22 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

Quoted 8 lines
Quoted 3 lines

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

Oh, where? Is that defined within the tests/docker/Makefile.include or in the TCG tree where I have not gone spelunking before?

--js

author
#10 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to Philippe Mathieu-Daudé
Quoted 47 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 23 lines · Alex Bennée
Quoted 22 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

Quoted 7 lines
Quoted 3 lines

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

For Sid I have this case:

Quoted 11 lines

-- >8 --
#
# Docker Renesas RX cross-compiler target
#
# This docker target builds on the debian Sid base image. #
# Copyright (c) 2019 Philippe Mathieu-Daudé
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
FROM qemu:debian-sid

Quoted 5 lines

Oh well nevermind it is old, so it probably now works with debian-10.

I used the Ports base for sparc32 and x32, I don't have them on my workstation I'll check during the WE.

If you want to keep docker-sid as a partial image for possible future uses, you might consider adding a simple amd64 user -- debian-sid-amd64 would suffice.

It doesn't have to do anything interesting, but it exists to give debian-sid a user.

I'd rather not waste your hard work, but I am making a plea that any images defined in the tree can be tested by running tests that use them.

--js

#12 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to John Snow
Quoted 30 lines · John Snow
Quoted 29 lines · Alex Bennée
Quoted 25 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Quoted 7 lines
Quoted 4 lines

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

Oh, where? Is that defined within the tests/docker/Makefile.include or in the TCG tree where I have not gone spelunking before?

Now they are selected by tests/tcg/configure.sh but generally they DISTRO-ARCH-cross images.

Quoted 2 lines

--js

--
Alex Bennée

author
#13 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

Quoted 38 lines · Alex Bennée
Quoted 35 lines · John Snow
Quoted 33 lines · Alex Bennée
Quoted 17 lines · John Snow
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 2 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

Well, I have WiP users from them. I could restore this content when they are ready... Ports is the base of deprecated Debian archs. On the other side Sid is the base for edge development I use from time to time to test latest gcc/binutils.
I'll try to find time to raise WiP branches to PoC.

I think that's the right thing to do. Right now, the docker tests directory has a lot of stale entries and unusable tests. That might be fine for the people working on it, but it makes it hard to understand and use for those of us who only occasionally traipse into the directory.

Quoted 8 lines

I'm removing all references to python2 -- but if there's no way for me to test debian-sid and debian-ports, I can't test changes I need to make to these "partial images", so they should be removed until they are consumable.

While I am sympathetic to the idea of having a library of partial images to use for future tests, they're prone to rot if there's no way to exercise them in-tree.

Don't forget some "partial" images are only used for building TCG tests
- we want to keep them. But as git is forever I can drop the sid/ports stuff for now until Phillipe has something to use them again.

Quoted 6 lines
Quoted 2 lines

Oh, where? Is that defined within the tests/docker/Makefile.include or in the TCG tree where I have not gone spelunking before?

Now they are selected by tests/tcg/configure.sh but generally they DISTRO-ARCH-cross images.

I see -- "container_image" in configure.sh chooses them and consumes them there.

Looks like that feeds to Makefile.qemu which defines docker-build-guest-tests.

We call docker.py (DOCKER_SCRIPT) as:
docker.py cc ... -i qemu:$(DOCKER_IMAGE) ...

okay, so this bypasses the Docker Makefile pretty directly; but it looks like it's only using images that the Makefile there doesn't count as "Partial" universally.

You *do* have a section that marks images as "PARTIAL" if they're the wrong architecture -- maybe it would be a semantic improvement to list these as simply "DISABLED" or "UNAVAILABLE" -- the implication being that they're complete images, but will not work for the current host architecture.

Thanks for the pointer!

--js

Reviewed-by on v1 3/3
#14 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

In reply to John Snow

Hi John,

Quoted 11 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker

Can you split this patch in 2, one per image? That will ease the reverts.

Once splitted feel free to add to each new patch:

Thanks,

Phil.

author
#15 lore raw

Re: [PATCH 3/3] docker: remove unused debian-sid and debian-ports

Quoted 15 lines · Philippe Mathieu-Daudé

Hi John,

Quoted 11 lines · John Snow

These are listed as "partial" images, but have no user. Remove them.

---
 tests/docker/Makefile.include                |  4 +--
 tests/docker/dockerfiles/debian-ports.docker | 36 --------------------
 tests/docker/dockerfiles/debian-sid.docker   | 35 -------------------
 3 files changed, 2 insertions(+), 73 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-ports.docker
 delete mode 100644 tests/docker/dockerfiles/debian-sid.docker
Quoted 9 lines

Can you split this patch in 2, one per image? That will ease the reverts.

Once splitted feel free to add to each new patch:

Thanks,

Phil.

Sounds good to me, thanks!