container images for migration-test

RFC Fabiano Rosas 5 participants 12 messages Sep 14 – Oct 15

Share & tools

Fabiano Rosas

author

2026-09-14 23:58 UTC #0 lore raw

Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.

From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:

$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB

This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:

CMD ["qemu-system-x86_64"]

The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:

QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
./tests/qtest/migration-test

(note that migration-test spawns the qemu binaries itself)

I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.

The reasons that lead me to this solution are:

1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.

2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

Any thoughts on this? Could it be useful to anyone? Maybe the stable branch has some similar issues?

I have a messy script that builds everything, but here are the important pieces:

- Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed to remove ncurses-devel (suse repo issue)

- Dockerfile.build.base ==> qemu-build-base:11.1.0 FROM qemu-rootfs-base:11.1.0

# actual source, mounted RO
RUN mkdir /src

# not the actual build dir, only the artifacts that are needed
RUN mkdir /build

WORKDIR /build-tmp-11.1.0

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
    /src/configure \
    --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
    --disable-plugins --disable-modules --enable-werror --enable-debug \
    --disable-docs --disable-tools && \
    make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
    cp -rL qemu-bundle /build && \
    cp -rL scripts /build && \
    find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; && \
    find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin \; && \
    find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec ln -sfT {} /bin/migration-test \; ;

---

- Dockerfile ==> qemu:11.1.0
FROM qemu-build-base:11.1.0 AS runtime

WORKDIR /build

# isolate the library dependencies of qemu, python and gdb
RUN mkdir /libs64 && \
    cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64 && \
    cp -L $(ldd /usr/lib64/$(readlink $(which python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
# rpm stuff above is because gdb has its own python runtime

RUN mkdir -p /slash/usr/share && \
    cp -r /usr/lib64/python* /libs64 && \
    ln -sfT /lib64 /slash/usr/lib64

RUN mkdir /libs && \
cp -r /usr/lib/rpm* /libs

# only these binaries in the image to keep it small
RUN mkdir /bins && \
    cp -L /bin/sh /bins/ && \
    cp -L /bin/ls /bins/ && \
    cp -L /bin/echo /bins/ && \
    cp -L /usr/bin/cat /bins/ && \
    cp -L /usr/bin/gdb /bins/ && \
    cp -L /usr/bin/python3 /bins/ && \
    cp -L /usr/bin/docker /bins/ && \
    cp -L /usr/bin/env /bins/ && \
    cp -rL /usr/share/gdb /slash/usr/share

# symlinks
RUN cp -d /bin/migration-test /bins/ && \
    cp -d /bin/qemu-system-* /bins/

FROM scratch
COPY --from=runtime /slash /
COPY --from=runtime /libs64 /lib64
COPY --from=runtime /libs /usr/lib
COPY --from=runtime /bins /bin
COPY --from=runtime /build /build
ENV PYTHONPATH="/lib64/python3.13"
ENV TMPDIR="/tmp/migtmp"
WORKDIR /build

# -version helps debugging, this is overriden when calling qemu with

# proper args
CMD ["qemu-system-x86_64", "-version"]
---

- The wrappers. One for each of qemu-system-x86_64, sh, migration-test, these live in a build-<version> directory.
---
#!/bin/sh

# pass the environment into the container 
echo > "/tmp/migtmp/envfile"
env | while IFS= read -r line; do
    case "$line" in
    QTEST*|QEMU*|PYTHON*)
        echo "$line" >> "/tmp/migtmp/envfile"
        ;;
    esac
done

bin=qemu-system-x86_64

exec docker run --rm -w /build \
                -v /tmp/migtmp:/tmp/migtmp \
                -v /home/farosas/archive:/archive \
                -v /var/run/docker.sock:/var/run/docker.sock \
                --network host \
                --env-file /tmp/migtmp/envfile \
                --cap-add=SYS_PTRACE \
                --security-opt seccomp=unconfined \
                --device /dev/kvm \
                --device /dev/userfaultfd:/dev/userfaultfd \
                -it qemu:11.1.0 $bin $(printf "%s\n" "$@")

---

- The snippet of the test script that uses all of this:

cd ./archive/build-11.0.0

# container-relative
CURRENT_QEMU=/archive/build-latest

if [ $1 = "src" ]; then

     info "forward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
else
     info "backward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}

fi
---

Peter Xu

2026-10-08 23:42 UTC #1 lore raw

Quoted 24 lines · Fabiano Rosas

Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
CMD ["qemu-system-x86_64"]
The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:
QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
./tests/qtest/migration-test

Quoted 5 lines

(note that migration-test spawns the qemu binaries itself)
I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.

I'm not a container person, e.g. I did test my recent channel fix on qemu-7.2 by simply building it with build-7.2/.. but after learning it the idea using a link to kickoff the container is very interesting on its own already.

Quoted 9 lines

The reasons that lead me to this solution are:
1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.

True.. One thing I want to make us on the same page is, this is about distro, not OS? Say, the kernel is still the same.

So we have three things (1) QEMU (2) distro (3) kernel/OS. Here the extra coverage is (2) only.

My gut feeling is distro specific thing causing migration failure is extremely rare; I can't even think of any in my work history. Meanwhile, it's still not the "real migration", because kernel is shared. Anything real will need to happen on two real and different hosts, ultimately, to make it cover all things. It's more likely to me that the kernel difference may fail a migration than distro difference.

So I still agree having such variance is better, though the hope is what we currently do e.g. with compat migration test is good enough.

Said that, if part of this RFC is to see if we want to introduce that too for the compat test, no objections either. Really, it's a matter of if it will help us find more bugs.

Quoted 6 lines

2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

This is definitely true. While studying your test approach, I also noticed that there's a "toolbox" tool that should have done many of what your script did: "enter" the toolbox will transparently map many things (files under /dev/, or $HOME) automatically. At least when thinking this as something I use personally, toolbox seems working. But again I'm not container person, maybe there's some difference, just FYI.

Quoted 3 lines

Any thoughts on this? Could it be useful to anyone? Maybe the stable branch has some similar issues?

I'd be surprised if Michael would need to test stable branch migrations! That's IMHO too much work for him.. if it needs to be more than what CI covers. After all, migration is only a submodule in stable maint pov.

So, I'm not sure whether some of those should be in upstream repo, let me know your thoughts. Besides, I'll definitely play more with toolbox and your scripts some day.

Thanks for sharing!

Quoted 141 lines

I have a messy script that builds everything, but here are the important pieces:

- Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed to remove ncurses-devel (suse repo issue)

- Dockerfile.build.base ==> qemu-build-base:11.1.0 FROM qemu-rootfs-base:11.1.0

# actual source, mounted RO
RUN mkdir /src

# not the actual build dir, only the artifacts that are needed
RUN mkdir /build

WORKDIR /build-tmp-11.1.0

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
    /src/configure \
    --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
    --disable-plugins --disable-modules --enable-werror --enable-debug \
    --disable-docs --disable-tools && \
    make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
    cp -rL qemu-bundle /build && \
    cp -rL scripts /build && \
    find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; && \
    find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin \; && \
    find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec ln -sfT {} /bin/migration-test \; ;

---

- Dockerfile ==> qemu:11.1.0
FROM qemu-build-base:11.1.0 AS runtime

WORKDIR /build

# isolate the library dependencies of qemu, python and gdb
RUN mkdir /libs64 && \
    cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64 && \
    cp -L $(ldd /usr/lib64/$(readlink $(which python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
# rpm stuff above is because gdb has its own python runtime

RUN mkdir -p /slash/usr/share && \
    cp -r /usr/lib64/python* /libs64 && \
    ln -sfT /lib64 /slash/usr/lib64

RUN mkdir /libs && \
cp -r /usr/lib/rpm* /libs

# only these binaries in the image to keep it small
RUN mkdir /bins && \
    cp -L /bin/sh /bins/ && \
    cp -L /bin/ls /bins/ && \
    cp -L /bin/echo /bins/ && \
    cp -L /usr/bin/cat /bins/ && \
    cp -L /usr/bin/gdb /bins/ && \
    cp -L /usr/bin/python3 /bins/ && \
    cp -L /usr/bin/docker /bins/ && \
    cp -L /usr/bin/env /bins/ && \
    cp -rL /usr/share/gdb /slash/usr/share

# symlinks
RUN cp -d /bin/migration-test /bins/ && \
    cp -d /bin/qemu-system-* /bins/

# -version helps debugging, this is overriden when calling qemu with

# proper args
CMD ["qemu-system-x86_64", "-version"]
---

- The wrappers. One for each of qemu-system-x86_64, sh, migration-test, these live in a build-<version> directory.
---
#!/bin/sh

# pass the environment into the container 
echo > "/tmp/migtmp/envfile"
env | while IFS= read -r line; do
    case "$line" in
    QTEST*|QEMU*|PYTHON*)
        echo "$line" >> "/tmp/migtmp/envfile"
        ;;
    esac
done

bin=qemu-system-x86_64

exec docker run --rm -w /build \
                -v /tmp/migtmp:/tmp/migtmp \
                -v /home/farosas/archive:/archive \
                -v /var/run/docker.sock:/var/run/docker.sock \
                --network host \
                --env-file /tmp/migtmp/envfile \
                --cap-add=SYS_PTRACE \
                --security-opt seccomp=unconfined \
                --device /dev/kvm \
                --device /dev/userfaultfd:/dev/userfaultfd \
                -it qemu:11.1.0 $bin $(printf "%s\n" "$@")

---

- The snippet of the test script that uses all of this:

cd ./archive/build-11.0.0

# container-relative
CURRENT_QEMU=/archive/build-latest

if [ $1 = "src" ]; then

     info "forward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
else
     info "backward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}

fi
---

Signature · 1 line

Peter Xu

Fabiano Rosas

author

2026-10-09 12:59 UTC #2 lore raw

Quoted 25 lines · Peter Xu

Quoted 24 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
CMD ["qemu-system-x86_64"]
The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:
QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
./tests/qtest/migration-test

Quoted 7 lines

Quoted 5 lines
(note that migration-test spawns the qemu binaries itself)
I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.
I'm not a container person,

Me neither, lol. Last container work I did was back when multiarch manifests were being added to docker. Maybe 2016? I (clearly) haven't kept up with the progress in the area.

Quoted 5 lines

e.g. I did test my recent channel fix on
qemu-7.2 by simply building it with build-7.2/.. but after learning it the idea using a link to kickoff the container is very interesting on its own already.

Yeah, the issue is that git checkout <old version> and try to build it is pretty hit or miss.

Quoted 10 lines

Quoted 9 lines
The reasons that lead me to this solution are:
1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.

Quoted 3 lines

True.. One thing I want to make us on the same page is, this is about distro, not OS? Say, the kernel is still the same.

Yes, it's about what it takes to get a working QEMU binary. I'm not trying to cover a realistic migration scenario, I'm simply trying to test across QEMU versions *somehow*.

Quoted 3 lines

So we have three things (1) QEMU (2) distro (3) kernel/OS. Here the extra coverage is (2) only.

It's not really about extra coverage, it's about (what I think should be) the minimum migration testing done by upstream, i.e. ensuring one binary we release is migration-compatible with another binary we release. Today we test n vs. n-1 in the CI and I think that's not enough.

Quoted 7 lines

My gut feeling is distro specific thing causing migration failure is extremely rare; I can't even think of any in my work history. Meanwhile, it's still not the "real migration", because kernel is shared. Anything real will need to happen on two real and different hosts, ultimately, to make it cover all things. It's more likely to me that the kernel difference may fail a migration than distro difference.

You right, but that feels like another problem for another day. It starts to get away from what the QEMU community can do. I've seen a lot of cpu-specific issues in the field actually, so even swapping kernels would miss a few bugs.

Quoted 7 lines

So I still agree having such variance is better, though the hope is what we currently do e.g. with compat migration test is good enough.
Said that, if part of this RFC is to see if we want to introduce that too for the compat test, no objections either. Really, it's a matter of if it will help us find more bugs.

Right, I'm not sure yet. Just sharing because I feel like keeping this stuff private is a bad approach. In general, if a maintainer relies on something to do their work, it should be under the community umbrella. Of course there are matters of _workflow_, and every person has a preference, but maybe the tooling could be shared.

Quoted 7 lines

Quoted 6 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

Quoted 7 lines

This is definitely true. While studying your test approach, I also noticed that there's a "toolbox" tool that should have done many of what your script did: "enter" the toolbox will transparently map many things (files under /dev/, or $HOME) automatically. At least when thinking this as something I use personally, toolbox seems working. But again I'm not container person, maybe there's some difference, just FYI.

I had no idea this existed, I'll take a look, thanks!

Quoted 8 lines

Quoted 3 lines
Any thoughts on this? Could it be useful to anyone? Maybe the stable branch has some similar issues?
I'd be surprised if Michael would need to test stable branch migrations! That's IMHO too much work for him.. if it needs to be more than what CI covers. After all, migration is only a submodule in stable maint pov.

Oh, not migration, but the idea of building an older codebase. I know he's dealt with distro repos being broken as well.

Quoted 7 lines

That's a good approach.
If you want to go further, you could directly host it on dockerhub, for debian or fedora hosts, so other don't need to rebuild anything. And everytime there is a new release, we would just add the new image.

It would save storage as well. I'm currently keeping a 7.2 build just because Peter found a bug in it.

Quoted 6 lines

$ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
podman run -it --rm \
-w $(pwd) -v $(pwd):$(pwd) \
docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
/usr/bin/qemu-system-aarch64 "$@"

Quoted 6 lines

You can drastically optimize size by using multi-stage builds. All images can be based on a common deps image, which contain all libraries qemu depend on (~140MB from what I can see on my debian system), and then all qemu-* images would be FROM:deps, and they only add a single binary inside.

Ah true, so I probably don't need the full distro build. And keeping the build around is also kinda pointless if it can just be rebuilt. I haven't yet encountered a scenario where a rebuild was necessary actually.

Quoted 4 lines

Final result would be 140MB for base deps + binary size per image on top, (from 3MB to 40MB) per binary. And we could have the whole collection of qemu binaries available for everyone with a single script.

Good to know, I haven't put too much effort in reducing the size of the images. Now I'm thinking that if they get quite small, we could also host a image just with the (non-unit) tests and maybe use it for some bisecting once a test fails.

Quoted 2 lines

Regards,
Pierrick

Alex Bennée

2026-10-09 15:13 UTC #3 lore raw

Quoted 58 lines · Fabiano Rosas

Quoted 25 lines · Peter Xu
Quoted 24 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
CMD ["qemu-system-x86_64"]
The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:
QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
./tests/qtest/migration-test
Quoted 7 lines
Quoted 5 lines
(note that migration-test spawns the qemu binaries itself)
I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.
I'm not a container person,
Me neither, lol. Last container work I did was back when multiarch manifests were being added to docker. Maybe 2016? I (clearly) haven't kept up with the progress in the area.
Quoted 5 lines
e.g. I did test my recent channel fix on
qemu-7.2 by simply building it with build-7.2/.. but after learning it the idea using a link to kickoff the container is very interesting on its own already.
Yeah, the issue is that git checkout <old version> and try to build it is pretty hit or miss.
Quoted 10 lines
Quoted 9 lines
The reasons that lead me to this solution are:
1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.

Quoted 7 lines

Quoted 3 lines
True.. One thing I want to make us on the same page is, this is about distro, not OS? Say, the kernel is still the same.
Yes, it's about what it takes to get a working QEMU binary. I'm not trying to cover a realistic migration scenario, I'm simply trying to test across QEMU versions *somehow*.

You could just consider using the system qemu if installed and then we get the variance between versions from which OS's we build on. That would work outside the container and we could include qemu-system in the containers as well.

It would cover the migrations from the various LTS qemu's to current master which should be the majority of common use cases.

Signature · 2 lines

Alex Bennée
Virtualisation Tech Lead @ Linaro

Pierrick Bouvier

2026-10-09 00:48 UTC #4 lore raw

In reply to Fabiano Rosas

Quoted 39 lines · Fabiano Rosas

Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
CMD ["qemu-system-x86_64"]
The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:
QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
./tests/qtest/migration-test
(note that migration-test spawns the qemu binaries itself)
I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.
The reasons that lead me to this solution are:
1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.

Quoted 9 lines

2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.
Any thoughts on this? Could it be useful to anyone? Maybe the stable branch has some similar issues?

That's a good approach.

If you want to go further, you could directly host it on dockerhub, for debian or fedora hosts, so other don't need to rebuild anything. And everytime there is a new release, we would just add the new image.

$ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
podman run -it --rm \
-w $(pwd) -v $(pwd):$(pwd) \
docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
/usr/bin/qemu-system-aarch64 "$@"

You can drastically optimize size by using multi-stage builds. All images can be based on a common deps image, which contain all libraries qemu depend on (~140MB from what I can see on my debian system), and then all qemu-* images would be FROM:deps, and they only add a single binary inside.

Final result would be 140MB for base deps + binary size per image on top, (from 3MB to 40MB) per binary. And we could have the whole collection of qemu binaries available for everyone with a single script.

Regards,
Pierrick

Pierrick Bouvier

2026-10-09 19:37 UTC #5 lore raw

See attached an example of a multi stage build that creates a container having only qemu binaries, their libraries, and qemu roms, and nothing else. It does not even have a shell.

build with:

$ podman build -f qemu.Dockerfile --build-arg QEMU_VERSION=v11.1.0 \
  -t test --squash-all

run with:

$ podman run -it --rm test qemu-system-aarch64 --version
QEMU emulator version 11.1.0 (v11.1.0)
Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers

Regards,
Pierrick

FROM debian:13 as build

RUN apt update && apt install -y git build-essential
RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/debian.sources RUN apt update && apt build-dep -y --arch-only qemu

ARG QEMU_VERSION
RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
RUN cd /tmp && \
git clone https://gitlab.com/qemu-project/qemu --depth 1 --single-branch --branch "${QEMU_VERSION}" && \
cd qemu && ./configure --prefix=/opt/qemu && \
ninja -C build install && \
rm -rf /tmp/*

# now retrieve runtime dependencies (libs + loader)
RUN ldd /opt/qemu/bin/* | grep '=>' | sed -e 's/.*=>//' -e 's/(.*//' | sort -u > /opt/qemu/libs.links
RUN find /lib/ld-linux* >> /opt/qemu/libs.links
RUN cat /opt/qemu/libs.links | xargs -n1 readlink -f > /opt/qemu/libs.realpath
# store them in /deps
RUN mkdir /deps && cp --parents $(cat /opt/qemu/libs.*) /deps/

################################################################################
FROM scratch
# FROM debian:13-slim

# in case you want a more complete container with a shell

# copy qemu and all associated artifacts (roms, ...)
COPY --from=build /opt/qemu /usr
COPY --from=build /deps /

# build with:
# podman build -f qemu.Dockerfile --build-arg QEMU_VERSION=v11.1.0 -t test --squash-all
# run with:
# podman run -it --rm test qemu-system-aarch64 --version

Pierrick Bouvier

2026-10-10 23:20 UTC #6 lore raw

In reply to Pierrick Bouvier

Quoted 68 lines · Pierrick Bouvier

Quoted 39 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
This last image is stripped (for size) of everything but a few libs and binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
CMD ["qemu-system-x86_64"]
The point is that I can wrap it in a script called qemu-system-x86_64 and run it interchangeably with any other qemu binary during migration-test:
  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
  ./tests/qtest/migration-test
(note that migration-test spawns the qemu binaries itself)
I run tests against the previous 3 QEMU releases. At every release I build the container for the new release and at every PR I build the container for that git hash using the latest tag as a base.
The reasons that lead me to this solution are:
1) Migration compat testing is special, the two migration peers can run in different OS versions, so their QEMU builds can be quite different. I cannot test using *just* my dev machine, neither using *just* a container. I need two hosts. Or two containers running different OSes that can both be reachable by the migration-test binary running in either of the OSes.
Quoted 9 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.
Any thoughts on this? Could it be useful to anyone? Maybe the stable branch has some similar issues?
That's a good approach.
If you want to go further, you could directly host it on dockerhub, for debian or fedora hosts, so other don't need to rebuild anything. And everytime there is a new release, we would just add the new image.
$ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
podman run -it --rm \
-w $(pwd) -v $(pwd):$(pwd) \
docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
/usr/bin/qemu-system-aarch64 "$@"
You can drastically optimize size by using multi-stage builds. All images can be based on a common deps image, which contain all libraries qemu depend on (~140MB from what I can see on my debian system), and then all qemu-* images would be FROM:deps, and they only add a single binary inside.

Quoted 4 lines

Final result would be 140MB for base deps + binary size per image on top, (from 3MB to 40MB) per binary. And we could have the whole collection of qemu binaries available for everyone with a single script.

I gave a try to this, see:
https://lore.kernel.org/qemu-devel/[email protected]/T/#t

Regards,
Pierrick

Daniel P. Berrangé

2026-10-12 08:19 UTC #7 lore raw

In reply to Fabiano Rosas

Quoted 10 lines · Fabiano Rosas

Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB

snip

Quoted 5 lines

2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

Yes, in general building old QEMU on new distros periodically hits trouble. If building old QEMU, better reliability is achieved by using a distro release that was from the same timeframe as QEMU. Is using containers targetting such old distro versions not an option ? The tests/docker/dockerfiles from the old QEMU branch ought to work ?

Our stable release branches have the same needs, and their CI is using the container images from the old branches.

How far back in time do you need/want to go for migration testing ? Is it sufficient to merely target versions for which mjt is still doing active maint work, so we can expect the containers to still work.

With regards,
Daniel

Signature · 3 lines

|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Pierrick Bouvier

2026-10-13 22:52 UTC #8 lore raw

Quoted 20 lines · Daniel P. Berrangé

Quoted 10 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
snip
Quoted 5 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

Quoted 7 lines

Yes, in general building old QEMU on new distros periodically hits trouble. If building old QEMU, better reliability is achieved by using a distro release that was from the same timeframe as QEMU. Is using containers targetting such old distro versions not an option ? The tests/docker/dockerfiles from the old QEMU branch ought to work ?

Using containers image we had in tree would be nice, assuming they still work today and don't reference any url that disappeared. I see that, for ubuntu for instance, we targeted some mid-LTS releases (like 19.04), so apt urls should be changed to https://old-releases.ubuntu.com/. It's quite hard to achieve true portability in dockerfiles over time.

As an alternative, this is an example of a single dockerfile that can build all our releases from 1.6.1 -> latest, using only LTS versions from 14.04 to 26.04:
https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 10 lines

Our stable release branches have the same needs, and their CI is using the container images from the old branches.
How far back in time do you need/want to go for migration testing ? Is it sufficient to merely target versions for which mjt is still doing active maint work, so we can expect the containers to still work.
With regards,
Daniel

Regards,
Pierrick

Fabiano Rosas

author

2026-10-14 17:57 UTC #9 lore raw

In reply to Daniel P. Berrangé

Quoted 20 lines · Daniel P. Berrangé

Quoted 10 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
snip
Quoted 5 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.

Quoted 7 lines

Yes, in general building old QEMU on new distros periodically hits trouble. If building old QEMU, better reliability is achieved by using a distro release that was from the same timeframe as QEMU. Is using containers targetting such old distro versions not an option ? The tests/docker/dockerfiles from the old QEMU branch ought to work ?

Yes, I'm doing just that. Some tweaks are needed still. For instance, my script has something like this:

dockerfile="tests/docker/dockerfiles/opensuse-leap.docker"

sed -i 's/ncurses-devel//' $dockerfile

if [ "$tag" = "10.1.0" ]; then

      sed -i 's/libLLVM19//' $dockerfile

Quoted 8 lines

Our stable release branches have the same needs, and their CI is using the container images from the old branches.
How far back in time do you need/want to go for migration testing ? Is it sufficient to merely target versions for which mjt is still doing active maint work, so we can expect the containers to still work.

I'm currently testing against the previous 3 releases. I think 2 releases is probably enough.

Quoted 2 lines

With regards,
Daniel

Daniel P. Berrangé

2026-10-15 07:18 UTC #10 lore raw

Quoted 33 lines · Fabiano Rosas

Quoted 20 lines · Daniel P. Berrangé
Quoted 10 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
snip
Quoted 5 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.
Quoted 7 lines
Yes, in general building old QEMU on new distros periodically hits trouble. If building old QEMU, better reliability is achieved by using a distro release that was from the same timeframe as QEMU. Is using containers targetting such old distro versions not an option ? The tests/docker/dockerfiles from the old QEMU branch ought to work ?
Yes, I'm doing just that. Some tweaks are needed still. For instance, my script has something like this:

Quoted 7 lines

dockerfile="tests/docker/dockerfiles/opensuse-leap.docker"
sed -i 's/ncurses-devel//' $dockerfile
if [ "$tag" = "10.1.0" ]; then
      sed -i 's/libLLVM19//' $dockerfile
fi

If that's needed, shouldn't we patch the QEMU stable branch ? I would have thought that would be a problem for doing releases on stable if the containers have bugs ?

Quoted 8 lines

Quoted 5 lines
How far back in time do you need/want to go for migration testing ? Is it sufficient to merely target versions for which mjt is still doing active maint work, so we can expect the containers to still work.
I'm currently testing against the previous 3 releases. I think 2 releases is probably enough.

I guess if we test X -> Y and then later test Y -> Z, then
we have transitively validated X -> Z. So in theory only 1 historical release is enough, but it makes sense to test one or two more variants just in case.

With regards,
Daniel

Signature · 3 lines

|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Fabiano Rosas

author

2026-10-15 12:46 UTC #11 lore raw

Quoted 38 lines · Daniel P. Berrangé

Quoted 37 lines · Fabiano Rosas
Quoted 20 lines · Daniel P. Berrangé
Quoted 10 lines · Fabiano Rosas
Hi, I just wanted to share and ask for feedback on some container images I've been building as part of my testing of migration pull-requests.
From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the following:
$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB
snip
Quoted 5 lines
2) Neither the QEMU build nor the distro repos are reliable enough to allow me to just "go back in time" and rebuild at any moment, I need to keep the artifacts cached to avoid rebuilding. Otherwise it comes time to send a PR and I discover that some python piece has changed or the distro messed up the older repository.
Quoted 7 lines
Yes, in general building old QEMU on new distros periodically hits trouble. If building old QEMU, better reliability is achieved by using a distro release that was from the same timeframe as QEMU. Is using containers targetting such old distro versions not an option ? The tests/docker/dockerfiles from the old QEMU branch ought to work ?
Yes, I'm doing just that. Some tweaks are needed still. For instance, my script has something like this:
dockerfile="tests/docker/dockerfiles/opensuse-leap.docker"
sed -i 's/ncurses-devel//' $dockerfile

Quoted 8 lines

Quoted 3 lines
if [ "$tag" = "10.1.0" ]; then
      sed -i 's/libLLVM19//' $dockerfile
fi
If that's needed, shouldn't we patch the QEMU stable branch ? I would have thought that would be a problem for doing releases on stable if the containers have bugs ?

Right, but that's not simply regular development work touching the dockerfiles and they sitting there just working. I see the overall benefits of having the dependencies listed in a dockerfile under our control, but for this particular use-case it's a bit of a pain point.

To be clear, this is not an issue I'm particularly concerned with, we can fix it in stable or hack away in a script like I did. And issues like this should be on the rarer side.

I'm more interested in the other part, which is having some infra that can take whatever base image and produce a container that's standin for the actual qemu binary. This allows quick switching of QEMU release versions for testing and debugging and keeps the migration testing concerns separate.

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines
How far back in time do you need/want to go for migration testing ? Is it sufficient to merely target versions for which mjt is still doing active maint work, so we can expect the containers to still work.

Quoted 8 lines

Quoted 2 lines
I'm currently testing against the previous 3 releases. I think 2 releases is probably enough.
I guess if we test X -> Y and then later test Y -> Z, then
we have transitively validated X -> Z. So in theory only 1 historical release is enough, but it makes sense to test one or two more variants just in case.

Yes, but sometimes we have weird compatibility knobs, I feel more comfortable with more testing. They're quick tests, it's not too much work. I usually run migration-test (small set) for every PR with:

x86_64: dev -> X, dev -> X-1, dev -> X-2

        dev <- X, dev <- X-1, dev <- X-2

s390x: dev -> X, dev -> X-1

       dev <- X, dev <- X-1

pcc64le: dev -> X
dev <- X

But there's also testing in the sense of working with bug reports, debugging and development in general. So having a few versions available improves productivity.

Quoted 2 lines

With regards,
Daniel