Skip to contentSkip to conversation

Yocto Gitlab CI

PATCH Superseded Bertrand Marquis 3 participants 9 messages Aug 24, 2022 – Oct 11, 2022

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover Yocto Gitlab CI

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is making sure build-yocto.sh is not catched by gitignore.

The second patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The third patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The fourth patch is adding a way to easily clean locally created containers.

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.

This has been tested on a x86 host machine and on an arm host machine (with mk_dsdt.c fix).

Bertrand Marquis (4):
automation: Only filter build-*/ in gitignore
automation: Add elements for Yocto test and run
automation: Add yocto containers with cache
automation: Add a clean rule for containers

 .gitignore                                    |   2 +-
 automation/build/Makefile                     |  19 +-
 automation/build/yocto/build-yocto.sh         | 328 ++++++++++++++++++
 .../build/yocto/kirkstone-qemuarm.dockerfile  |  28 ++
 .../yocto/kirkstone-qemuarm64.dockerfile      |  28 ++
 .../yocto/kirkstone-qemux86-64.dockerfile     |  28 ++
 automation/build/yocto/kirkstone.dockerfile   | 100 ++++++
 7 files changed, 531 insertions(+), 2 deletions(-)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone.dockerfile
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/4 automation: Only filter build-*/ in gitignore

...because there are some script in automation corresponding to the build-* filter (build-test.sh and build-each-commit.sh)

---
Changes in v1:
- instead of removing build-*, replace it by build-*/ (suggested by Jan)
- rename patch accordingly

---
 .gitignore | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitignore +1 -1
index e3fe06cd13ed..27881c976432 100644
@@ -38,13 +38,13 @@ dist
 stubdom/*.tar.gz
 
 autom4te.cache/
+build-*/
 config.log
 config.status
 config.cache
 config/Toplevel.mk
 config/Paths.mk
 
-build-*
 dist/*
 docs/tmp.*
 docs/html/
Signature · 1 line
2.25.1
#4 lore raw

PATCH 3/4 automation: Add yocto containers with cache

In reply to Bertrand Marquis

Create yocto containers for kirkstone with a builder step to fill the yocto cache by running the build script once.
Those containers depend on the kirkstone.dockerfile container to be created and use it as base image.

To run a build test on one image the following command must be called inside the container:
build-yocto.sh --xen-dir=XENSRC TARGET where
- XENSRC is the directory where xen sources are available in the container
- TARGET is the build target (qemuarm64, qemuarm or qemux86_64)

In case of error, the script will return an error code and the logs will be in /home/docker-build/logs.

!!!
The build of each container can take several hours depending on the network bandwith (for downloading all sources) and the amount of computing power on the machine where the docker images are created. !!!

---
Changes in v1:
- fix dependency in Makefile

---
 automation/build/Makefile                     |  2 ++
 .../build/yocto/kirkstone-qemuarm.dockerfile  | 28 +++++++++++++++++++
 .../yocto/kirkstone-qemuarm64.dockerfile      | 28 +++++++++++++++++++
 .../yocto/kirkstone-qemux86-64.dockerfile     | 28 +++++++++++++++++++
 4 files changed, 86 insertions(+)
 create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
automation/build/Makefile +2 -0
index a4b2b85178cf..b0dc68f00b70 100644
@@ -16,5 +16,7 @@ help:
 		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
+$(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone
+
 .PHONY: all
 all: $(CONTAINERS)
automation/build/yocto/kirkstone-qemuarm.dockerfile new file +28 -0
new file mode 100644
index 000000000000..e3cdc4dcf333
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemuarm
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
automation/build/yocto/kirkstone-qemuarm64.dockerfile new file +28 -0
new file mode 100644
index 000000000000..6045b6ac249f
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemuarm64
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
automation/build/yocto/kirkstone-qemux86-64.dockerfile new file +28 -0
new file mode 100644
index 000000000000..35e2fdeb1e82
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemux86-64
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
Signature · 1 line
2.25.1
#5 lore raw

PATCH 4/4 automation: Add a clean rule for containers

In reply to Bertrand Marquis

Add make clean support to remove the containers from the local docker registry:
- make clean: remove all images
- clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm image

---
Changes in v1:
- patch added

---
 automation/build/Makefile | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
automation/build/Makefile +16 -1
index b0dc68f00b70..9fbb49c8717e 100644
@@ -4,6 +4,17 @@ REGISTRY := registry.gitlab.com/xen-project/xen
 CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
 DOCKER_CMD ?= docker
 
+define CLEAN_RULE
+.PHONY: clean-$(1)
+clean-$(1):
+ifneq ($$(shell docker image ls -q $(REGISTRY)/$(subst /,:,$(1))),)
+	docker image rm $(REGISTRY)/$(subst /,:,$(1))
+endif
+
+clean: clean-$(1)
+
+endef
+
 help:
 	@echo "Builds containers for building Xen based on different distros"
 	@echo "To build one run 'make DISTRO/VERSION'. Available containers:"
@@ -18,5 +29,9 @@ help:
 
 $(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone
 
-.PHONY: all
+.PHONY: all clean
 all: $(CONTAINERS)
+
+clean:
+
+$(eval $(foreach img,$(CONTAINERS),$(call CLEAN_RULE,$(img))))
Signature · 1 line
2.25.1
#3 lore raw

PATCH 2/4 automation: Add elements for Yocto test and run

In reply to Bertrand Marquis

Add a container suitable to run yocto kirkstone build based on ubuntu 22.04. It contains all packages required by Yocto and a checkout of the layers required to build Xen with Yocto.

Add a build script to build and run xen on qemu using Yocto. The script supports arm32, arm64 and x86_64 and checks that dom0 is properly booting. At this stage this does not run any guest on top of dom0.

The build script is currently in the same directory than the docker files so that it can be included directly in the image. This is required for the next step which creates images with an initial cache content to speed up testing.

---
Changes in v1:
- add --image command line argument to build-yocto.sh to allow building something different than xen-image-minimal.
- modify dockerfile to have one layer per line and make it easier to add other. I kept the for loop to keep the number of docker steps lower
- update commit message to warn that no guest are tested.
- fix build-yocto script to properly return with an error if expect script ends up in timeout or EOF.

---
 automation/build/yocto/build-yocto.sh       | 328 ++++++++++++++++++++
 automation/build/yocto/kirkstone.dockerfile | 100 ++++++
 2 files changed, 428 insertions(+)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/kirkstone.dockerfile
automation/build/yocto/build-yocto.sh new file +328 -0
new file mode 100755
index 000000000000..669359daa52e
@@ -0,0 +1,328 @@
+#!/bin/bash
+#
+# Yocto meta virtualization build and run script
+#
+# This script is building Yocto xen-image-minimal for qemu targets and run
+# them using runqemu inside yocto to check that dom0 is booting properly
+# The build is using a local xen source tree so that specific patches can be
+# tested.
+# In order to optimize the build time, a build cache is used so that only xen
+# packages and its dependencies are rebuilt (qemu and final image mainly).
+#
+
+# Directories
+YOCTODIR="$HOME/yocto-layers"
+CACHEDIR="$HOME/yocto-cache"
+LOGDIR="$HOME/logs"
+XENDIR="$HOME/xen"
+BUILDDIR="$HOME/build"
+
+# what yocto bsp we support
+TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
+VERBOSE="n"
+TARGETLIST=""
+
+# actions to do
+do_clean="n"
+do_build="y"
+do_run="y"
+do_localsrc="n"
+build_result=0
+
+# layers to include in the project
+build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
+                 meta-openembedded/meta-oe meta-openembedded/meta-python \
+                 meta-openembedded/meta-filesystems \
+                 meta-openembedded/meta-networking meta-virtualization"
+
+# yocto image to build
+build_image="xen-image-minimal"
+
+function print_progress() {
+    echo -n "$(date +%T) $*"
+}
+
+function run_task() {
+    local task_name="$1"
+    local task_target="$2"
+
+    task_log="${task_name//project_}-${task_target}"
+
+    mkdir -p "${LOGDIR}"
+    print_progress
+    echo -n "${task_name//project_} ${task_target}: "
+    if [ "${VERBOSE}" = "n" ]; then
+        $@ > "${LOGDIR}/${task_log}.log" 2>&1
+    else
+        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
+    fi
+
+    if [ ${?} -ne 0 ]; then
+        echo "Error"
+        build_result=$((build_result+1))
+        return 1
+    else
+        echo "OK"
+        return 0
+    fi
+}
+
+function project_create() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        # init yocto project
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        # add needed layers
+        for layer in ${build_layerlist}; do
+            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
+        done
+    ) || return 1
+
+    # customize project configuration
+    cat <<EOF >> "${destdir}/conf/local.conf"
+# Yocto BSP
+MACHINE = "${target}"
+
+# Use local cache to reuse previous builds results
+SSTATE_DIR = "${CACHEDIR}/sstate-cache"
+DL_DIR = "${CACHEDIR}/downloads"
+
+# Enable xen and virtualization
+DISTRO_FEATURES = " virtualization xen ipv4"
+
+# Speed up run by not generating ssh host keys
+IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
+
+# Save some disk space
+INHERIT += "rm_work"
+
+EOF
+
+    if [ "${do_localsrc}" = "y" ]; then
+        XENVERS=$(grep -e "^XEN_REL" \
+            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
+            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
+            | sed -e "s/.*=//" | sort -V | tail -n 1)
+
+        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
+        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
+
+        cat <<EOF >> "${destdir}/conf/local.conf"
+# Use local sources for xen and xen-tools
+FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
+FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
+
+SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
+SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
+
+PREFERRED_VERSION:pn-xen = "${XENVERS}%"
+PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
+
+S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
+S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
+
+SRCREV:pn-xen = "\${AUTOREV}"
+SRCREV:pn-xen-tools = "\${AUTOREV}"
+
+SRCPV:pn-xen = "1"
+SRCPV:pn-xen-tools = "1"
+
+# Disable all QA errors as the recipe is not up to date with changes in Xen
+# when we use local sources
+ERROR_QA:pn-xen = "arch"
+ERROR_QA:pn-xen-tools = "arch"
+
+EOF
+    fi
+}
+
+function project_build() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        bitbake "${build_image}" || exit 1
+    ) || return 1
+}
+
+function project_clean() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    rm -rf "${destdir}"
+}
+
+function project_run() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
+
+        /usr/bin/expect <<EOF
+set timeout 100
+spawn bash -c "runqemu serialstdio nographic slirp"
+
+expect_after {
+    -re "(.*)\r" {
+        exp_continue
+    }
+    timeout {send_user "ERROR-Timeout!\n"; exit 1}
+    eof {send_user "ERROR-EOF!\n"; exit 1}
+}
+
+# wait initial login
+expect -re ".* login: "
+send "root\r"
+expect -re "root@.*# "
+
+EOF
+    exit $?
+    ) || return 1
+}
+
+function help() {
+    cat <<EOF
+Usage: ${0} [TARGET1] [TARGET2]
+
+This script is build the yocto xen-image-minimal for different qemu targets
+and is running it after.
+Without any target specified, all supported targets are done.
+
+Options:
+  -h, --help       Print this help
+  -v, --verbose    Verbose build
+  --list-target    List supported targets
+  --clean          Clean existing project before starting
+  --no-build       Do not build (to run an already built project)
+  --no-run         Do not run
+  --image=IMG      Yocto image or package to build
+                   Default: xen-image-minimal
+  --xen-dir=DIR    path to xen hypervisor source tree
+                   if not provide, normal yocto version of xen is built
+                   Default: ${XENDIR}
+  --out-dir=DIR    directory where to create the projectss
+                   Default: ${BUILDDIR}
+  --log-dir=DIR    directory to store logs
+                   Default: ${LOGDIR}
+  --cache-dir=DIR  directory where to take and store build cache
+                   Default: ${CACHEDIR}
+  --layer-dir=DIR  directory containing the checkout of yocto layers
+                   Default: ${YOCTODIR}
+EOF
+}
+
+for OPTION in "$@"
+do
+    case ${OPTION} in
+        -h|--help)
+            help
+            exit 0
+            ;;
+        -v|--verbose)
+            VERBOSE="y"
+            ;;
+        --list-targets)
+            echo "${TARGET_SUPPORTED}"
+            exit 0
+            ;;
+        --clean)
+            do_clean="y"
+            ;;
+        --no-build)
+            do_build="n"
+            ;;
+        --no-run)
+            do_run="n"
+            ;;
+        --image=*)
+            build_image="${OPTION#*=}"
+            ;;
+        --xen-dir=*)
+            XENDIR="${OPTION#*=}"
+            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
+                echo "No Xen source tree in ${XENDIR}"
+                exit 1
+            fi
+            do_localsrc="y"
+            ;;
+        --out-dir=*)
+            BUILDDIR="${OPTION#*=}"
+            ;;
+        --log-dir=*)
+            LOGDIR="${OPTION#*=}"
+            ;;
+        --cache-dir=*)
+            CACHEDIR="${OPTION#*=}"
+            ;;
+        --layer-dir=*)
+            YOCTODIR="${OPTION#*=}"
+            ;;
+        --*)
+            echo "Invalid option ${OPTION}"
+            help
+            exit 1
+            ;;
+        *)
+            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
+                TARGETLIST="${TARGETLIST} ${OPTION}"
+            else
+                echo "Unsupported target ${OPTION}"
+                exit 1
+            fi
+            ;;
+    esac
+done
+
+# if no target is specified build all targets
+if [ -z "${TARGETLIST}" ]; then
+    TARGETLIST="${TARGET_SUPPORTED}"
+fi
+
+mkdir -p "${CACHEDIR}"
+mkdir -p "${LOGDIR}"
+mkdir -p "${BUILDDIR}"
+
+# Make sure we have an absolute path
+YOCTODIR=$(realpath -m "${YOCTODIR}")
+CACHEDIR=$(realpath -m "${CACHEDIR}")
+BUILDDIR=$(realpath -m "${BUILDDIR}")
+LOGDIR=$(realpath -m "${LOGDIR}")
+if [ "${do_localsrc}" = "y" ]; then
+    XENDIR=$(realpath -m "${XENDIR}")
+fi
+
+# Check that we have all the layers we need
+for f in ${build_layerlist}; do
+    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
+        echo "Layer ${f} missing in ${YOCTODIR}"
+        exit 1
+    fi
+done
+
+for f in ${TARGETLIST}; do
+    if [ "${do_clean}" = "y" ]; then
+        run_task project_clean ${f}
+    fi
+    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        run_task project_create ${f}
+    fi
+    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        if [ "${do_build}" = "y" ]; then
+            run_task project_build ${f}
+        fi
+        if [ "${do_run}" = "y" ]; then
+            run_task project_run ${f}
+        fi
+
+    fi
+done
+
+print_progress "Build Complete (${build_result} errors)"
+echo
+exit ${build_result}
+
automation/build/yocto/kirkstone.dockerfile new file +100 -0

100 lines not on this page · raw message

Signature · 1 line
2.25.1
#6 lore raw
In reply to Bertrand Marquis
Quoted 17 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is making sure build-yocto.sh is not catched by gitignore.

The second patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The third patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The fourth patch is adding a way to easily clean locally created containers.

Quoted 13 lines

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

So I tried to build one of the build containers on my x86 workstation with the following:

make yocto/kirkstone-qemuarm64

but I get an error from the build:

21:30:20 build qemuarm64: Error
22:00:38 run qemuarm64: Error
22:00:41 Build Complete (2 errors)
The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2

Anyone else is having a better luck than me?

I don't think it is a problem if it takes a long time to build the build containers because they are not built often and they are not built as part of the gitlab-ci runs.

The issue could be the resulting container size. I wasn't aware of a limit in gitlab -- I would like to try if there is a way around the limit (either by changing a setting, or potentially switching to a premium account.) However I need to be able to complete a container build first :-)

How did you find out about the 20 GB limit? I couldn't find it in the docs. The only info I could find states that there is no hard limit on registry.gitlab.com.

Cheers,

Stefano

#7 lore raw

HI Stefano,

Quoted 39 lines · Stefano Stabellini
Quoted 17 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is making sure build-yocto.sh is not catched by gitignore.

The second patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The third patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The fourth patch is adding a way to easily clean locally created containers.

Quoted 13 lines

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

So I tried to build one of the build containers on my x86 workstation with the following:

make yocto/kirkstone-qemuarm64

Quoted 8 lines

but I get an error from the build:

21:30:20 build qemuarm64: Error
22:00:38 run qemuarm64: Error
22:00:41 Build Complete (2 errors)
The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2

Anyone else is having a better luck than me?

I will relaunch an image creation test locally and come back to you. It could be that Yocto has been updated since I last tested that.

Quoted 12 lines

I don't think it is a problem if it takes a long time to build the build containers because they are not built often and they are not built as part of the gitlab-ci runs.

The issue could be the resulting container size. I wasn't aware of a limit in gitlab -- I would like to try if there is a way around the limit (either by changing a setting, or potentially switching to a premium account.) However I need to be able to complete a container build first :-)

Quoted 3 lines

How did you find out about the 20 GB limit? I couldn't find it in the docs. The only info I could find states that there is no hard limit on registry.gitlab.com.

This came after discussions internally with our internal gitlab and I found some forum discussions saying something like that.
But it could be that Xen Project does not have this limitation hence the need to test.

Cheers
Bertrand

Quoted 4 lines

Cheers,

Stefano

#8 lore raw
In reply to Stefano Stabellini

Hi Stefano,

Quoted 39 lines · Stefano Stabellini
Quoted 17 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is making sure build-yocto.sh is not catched by gitignore.

The second patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The third patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The fourth patch is adding a way to easily clean locally created containers.

Quoted 13 lines

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

So I tried to build one of the build containers on my x86 workstation with the following:

make yocto/kirkstone-qemuarm64

Quoted 9 lines

but I get an error from the build:

21:30:20 build qemuarm64: Error
22:00:38 run qemuarm64: Error
22:00:41 Build Complete (2 errors)
The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2

Anyone else is having a better luck than me?

I did a new run and everything went ok on my side.
I will push a v2 of the serie to dump more logs when an error is happening.

Cheers
Bertrand

Quoted 18 lines

I don't think it is a problem if it takes a long time to build the build containers because they are not built often and they are not built as part of the gitlab-ci runs.

The issue could be the resulting container size. I wasn't aware of a limit in gitlab -- I would like to try if there is a way around the limit (either by changing a setting, or potentially switching to a premium account.) However I need to be able to complete a container build first :-)

How did you find out about the 20 GB limit? I couldn't find it in the docs. The only info I could find states that there is no hard limit on registry.gitlab.com.

Cheers,

Stefano