Skip to contentSkip to conversation

Yocto Gitlab CI

PATCH Superseded Bertrand Marquis 3 participants 14 messages Oct 12, 2022 – Oct 19, 2022

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover Yocto Gitlab CI

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The third patch is adding a way to easily clean locally created containers.

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.

This has been tested on a x86 host machine and on an arm host machine (with mk_dsdt.c fix).

Changes in v2:
- remove gitignore patch which was merged
- add a --dump-log support in build-yocto.sh script and use it during container creation to see the error logs.

Bertrand Marquis (3):
automation: Add elements for Yocto test and run
automation: Add yocto containers with cache
automation: Add a clean rule for containers

 automation/build/Makefile                     |  19 +-
 automation/build/yocto/build-yocto.sh         | 340 ++++++++++++++++++
 .../build/yocto/kirkstone-qemuarm.dockerfile  |  28 ++
 .../yocto/kirkstone-qemuarm64.dockerfile      |  28 ++
 .../yocto/kirkstone-qemux86-64.dockerfile     |  28 ++
 automation/build/yocto/kirkstone.dockerfile   | 100 ++++++
 6 files changed, 542 insertions(+), 1 deletion(-)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone.dockerfile
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/3 automation: Add elements for Yocto test and run

Add a container suitable to run yocto kirkstone build based on ubuntu 22.04. It contains all packages required by Yocto and a checkout of the layers required to build Xen with Yocto.

Add a build script to build and run xen on qemu using Yocto. The script supports arm32, arm64 and x86_64 and checks that dom0 is properly booting. At this stage this does not run any guest on top of dom0.

The build script is currently in the same directory than the docker files so that it can be included directly in the image. This is required for the next step which creates images with an initial cache content to speed up testing.

---
Changes in v2:
- add a --dump-log command line option to build-yocto.sh script to dump the logs if an error occurs.
Changes in v1:
- add --image command line argument to build-yocto.sh to allow building something different than xen-image-minimal.
- modify dockerfile to have one layer per line and make it easier to add other. I kept the for loop to keep the number of docker steps lower
- update commit message to warn that no guest are tested.
- fix build-yocto script to properly return with an error if expect script ends up in timeout or EOF.

---
 automation/build/yocto/build-yocto.sh       | 340 ++++++++++++++++++++
 automation/build/yocto/kirkstone.dockerfile | 100 ++++++
 2 files changed, 440 insertions(+)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/kirkstone.dockerfile
automation/build/yocto/build-yocto.sh new file +340 -0
new file mode 100755
index 000000000000..0d31dad607fb
@@ -0,0 +1,340 @@
+#!/bin/bash
+#
+# Yocto meta virtualization build and run script
+#
+# This script is building Yocto xen-image-minimal for qemu targets and run
+# them using runqemu inside yocto to check that dom0 is booting properly
+# The build is using a local xen source tree so that specific patches can be
+# tested.
+# In order to optimize the build time, a build cache is used so that only xen
+# packages and its dependencies are rebuilt (qemu and final image mainly).
+#
+
+# Directories
+YOCTODIR="$HOME/yocto-layers"
+CACHEDIR="$HOME/yocto-cache"
+LOGDIR="$HOME/logs"
+XENDIR="$HOME/xen"
+BUILDDIR="$HOME/build"
+
+# what yocto bsp we support
+TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
+VERBOSE="n"
+TARGETLIST=""
+
+# actions to do
+do_clean="n"
+do_build="y"
+do_run="y"
+do_localsrc="n"
+do_dump="n"
+build_result=0
+
+# layers to include in the project
+build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
+                 meta-openembedded/meta-oe meta-openembedded/meta-python \
+                 meta-openembedded/meta-filesystems \
+                 meta-openembedded/meta-networking meta-virtualization"
+
+# yocto image to build
+build_image="xen-image-minimal"
+
+function print_progress() {
+    echo -n "$(date +%T) $*"
+}
+
+function run_task() {
+    local task_name="$1"
+    local task_target="$2"
+
+    task_log="${task_name//project_}-${task_target}"
+
+    mkdir -p "${LOGDIR}"
+    print_progress
+    echo -n "${task_name//project_} ${task_target}: "
+    if [ "${VERBOSE}" = "n" ]; then
+        $@ > "${LOGDIR}/${task_log}.log" 2>&1
+    else
+        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
+    fi
+
+    if [ ${?} -ne 0 ]; then
+        echo "Error"
+        build_result=$((build_result+1))
+        if [ "${do_dump}" = "y" ]; then
+            echo
+            echo "############ LOGS-START ############"
+            cat "${LOGDIR}/${task_log}.log"
+            echo "############  LOGS-END  ############"
+            echo
+        fi
+        return 1
+    else
+        echo "OK"
+        return 0
+    fi
+}
+
+function project_create() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        # init yocto project
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        # add needed layers
+        for layer in ${build_layerlist}; do
+            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
+        done
+    ) || return 1
+
+    # customize project configuration
+    cat <<EOF >> "${destdir}/conf/local.conf"
+# Yocto BSP
+MACHINE = "${target}"
+
+# Use local cache to reuse previous builds results
+SSTATE_DIR = "${CACHEDIR}/sstate-cache"
+DL_DIR = "${CACHEDIR}/downloads"
+
+# Enable xen and virtualization
+DISTRO_FEATURES = " virtualization xen ipv4"
+
+# Speed up run by not generating ssh host keys
+IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
+
+# Save some disk space
+INHERIT += "rm_work"
+
+EOF
+
+    if [ "${do_localsrc}" = "y" ]; then
+        XENVERS=$(grep -e "^XEN_REL" \
+            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
+            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
+            | sed -e "s/.*=//" | sort -V | tail -n 1)
+
+        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
+        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
+
+        cat <<EOF >> "${destdir}/conf/local.conf"
+# Use local sources for xen and xen-tools
+FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
+FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
+
+SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
+SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
+
+PREFERRED_VERSION:pn-xen = "${XENVERS}%"
+PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
+
+S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
+S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
+
+SRCREV:pn-xen = "\${AUTOREV}"
+SRCREV:pn-xen-tools = "\${AUTOREV}"
+
+SRCPV:pn-xen = "1"
+SRCPV:pn-xen-tools = "1"
+
+# Disable all QA errors as the recipe is not up to date with changes in Xen
+# when we use local sources
+ERROR_QA:pn-xen = "arch"
+ERROR_QA:pn-xen-tools = "arch"
+
+EOF
+    fi
+}
+
+function project_build() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        bitbake "${build_image}" || exit 1
+    ) || return 1
+}
+
+function project_clean() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    rm -rf "${destdir}"
+}
+
+function project_run() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
+
+        /usr/bin/expect <<EOF
+set timeout 100
+spawn bash -c "runqemu serialstdio nographic slirp"
+
+expect_after {
+    -re "(.*)\r" {
+        exp_continue
+    }
+    timeout {send_user "ERROR-Timeout!\n"; exit 1}
+    eof {send_user "ERROR-EOF!\n"; exit 1}
+}
+
+# wait initial login
+expect -re ".* login: "
+send "root\r"
+expect -re "root@.*# "
+
+EOF
+    exit $?
+    ) || return 1
+}
+
+function help() {
+    cat <<EOF
+Usage: ${0} [TARGET1] [TARGET2]
+
+This script is build the yocto xen-image-minimal for different qemu targets
+and is running it after.
+Without any target specified, all supported targets are done.
+
+Options:
+  -h, --help       Print this help
+  -v, --verbose    Verbose build
+  --list-target    List supported targets
+  --clean          Clean existing project before starting
+  --no-build       Do not build (to run an already built project)
+  --no-run         Do not run
+  --dump-log       On error, dump the logs on the console
+  --image=IMG      Yocto image or package to build
+                   Default: xen-image-minimal
+  --xen-dir=DIR    path to xen hypervisor source tree
+                   if not provide, normal yocto version of xen is built
+                   Default: ${XENDIR}
+  --out-dir=DIR    directory where to create the projectss
+                   Default: ${BUILDDIR}
+  --log-dir=DIR    directory to store logs
+                   Default: ${LOGDIR}
+  --cache-dir=DIR  directory where to take and store build cache
+                   Default: ${CACHEDIR}
+  --layer-dir=DIR  directory containing the checkout of yocto layers
+                   Default: ${YOCTODIR}
+EOF
+}
+
+for OPTION in "$@"
+do
+    case ${OPTION} in
+        -h|--help)
+            help
+            exit 0
+            ;;
+        -v|--verbose)
+            VERBOSE="y"
+            ;;
+        --list-targets)
+            echo "${TARGET_SUPPORTED}"
+            exit 0
+            ;;
+        --clean)
+            do_clean="y"
+            ;;
+        --no-build)
+            do_build="n"
+            ;;
+        --no-run)
+            do_run="n"
+            ;;
+        --dump-log)
+            do_dump="y"
+            ;;
+        --image=*)
+            build_image="${OPTION#*=}"
+            ;;
+        --xen-dir=*)
+            XENDIR="${OPTION#*=}"
+            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
+                echo "No Xen source tree in ${XENDIR}"
+                exit 1
+            fi
+            do_localsrc="y"
+            ;;
+        --out-dir=*)
+            BUILDDIR="${OPTION#*=}"
+            ;;
+        --log-dir=*)
+            LOGDIR="${OPTION#*=}"
+            ;;
+        --cache-dir=*)
+            CACHEDIR="${OPTION#*=}"
+            ;;
+        --layer-dir=*)
+            YOCTODIR="${OPTION#*=}"
+            ;;
+        --*)
+            echo "Invalid option ${OPTION}"
+            help
+            exit 1
+            ;;
+        *)
+            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
+                TARGETLIST="${TARGETLIST} ${OPTION}"
+            else
+                echo "Unsupported target ${OPTION}"
+                exit 1
+            fi
+            ;;
+    esac
+done
+
+# if no target is specified build all targets
+if [ -z "${TARGETLIST}" ]; then
+    TARGETLIST="${TARGET_SUPPORTED}"
+fi
+
+mkdir -p "${CACHEDIR}"
+mkdir -p "${LOGDIR}"
+mkdir -p "${BUILDDIR}"
+
+# Make sure we have an absolute path
+YOCTODIR=$(realpath -m "${YOCTODIR}")
+CACHEDIR=$(realpath -m "${CACHEDIR}")
+BUILDDIR=$(realpath -m "${BUILDDIR}")
+LOGDIR=$(realpath -m "${LOGDIR}")
+if [ "${do_localsrc}" = "y" ]; then
+    XENDIR=$(realpath -m "${XENDIR}")
+fi
+
+# Check that we have all the layers we need
+for f in ${build_layerlist}; do
+    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
+        echo "Layer ${f} missing in ${YOCTODIR}"
+        exit 1
+    fi
+done
+
+for f in ${TARGETLIST}; do
+    if [ "${do_clean}" = "y" ]; then
+        run_task project_clean ${f}
+    fi
+    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        run_task project_create ${f}
+    fi
+    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        if [ "${do_build}" = "y" ]; then
+            run_task project_build ${f}
+        fi
+        if [ "${do_run}" = "y" ]; then
+            run_task project_run ${f}
+        fi
+
+    fi
+done
+
+print_progress "Build Complete (${build_result} errors)"
+echo
+exit ${build_result}
+
automation/build/yocto/kirkstone.dockerfile new file +100 -0

100 lines not on this page · raw message

Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/3 automation: Add yocto containers with cache

In reply to Bertrand Marquis

Create yocto containers for kirkstone with a builder step to fill the yocto cache by running the build script once.
Those containers depend on the kirkstone.dockerfile container to be created and use it as base image.

To run a build test on one image the following command must be called inside the container:
build-yocto.sh --xen-dir=XENSRC TARGET where
- XENSRC is the directory where xen sources are available in the container
- TARGET is the build target (qemuarm64, qemuarm or qemux86_64)

In case of error, the script will return an error code and the logs will be in /home/docker-build/logs.

!!!
The build of each container can take several hours depending on the network bandwith (for downloading all sources) and the amount of computing power on the machine where the docker images are created. !!!

---
Changes in v2:
- pass --dump-log option to build-yocto.sh during the image generation so that we have more info when there is an error during the image creation.
Changes in v1:
- fix dependency in Makefile

---
 automation/build/Makefile                     |  2 ++
 .../build/yocto/kirkstone-qemuarm.dockerfile  | 28 +++++++++++++++++++
 .../yocto/kirkstone-qemuarm64.dockerfile      | 28 +++++++++++++++++++
 .../yocto/kirkstone-qemux86-64.dockerfile     | 28 +++++++++++++++++++
 4 files changed, 86 insertions(+)
 create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
automation/build/Makefile +2 -0
index a4b2b85178cf..b0dc68f00b70 100644
@@ -16,5 +16,7 @@ help:
 		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
+$(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone
+
 .PHONY: all
 all: $(CONTAINERS)
automation/build/yocto/kirkstone-qemuarm.dockerfile new file +28 -0
new file mode 100644
index 000000000000..b6bd8ceee9a7
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemuarm
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
automation/build/yocto/kirkstone-qemuarm64.dockerfile new file +28 -0
new file mode 100644
index 000000000000..f279a7af928c
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemuarm64
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
automation/build/yocto/kirkstone-qemux86-64.dockerfile new file +28 -0
new file mode 100644
index 000000000000..595643df5905
@@ -0,0 +1,28 @@
+# Docker file to populate the initial sstate and download cache for yocto
+# for a specific target
+#
+# This requires the build-yocto.sh script to be in the same directory as this
+# dockerfile during the image creation
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "from_image=img" to select the docker image and tag to start from
+# "target=qemuxxx": to select which target to build in this image
+
+ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone
+
+FROM $from_image AS builder
+
+ARG target=qemux86-64
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+
+FROM $from_image
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="[email protected]"
+
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 3/3 automation: Add a clean rule for containers

In reply to Bertrand Marquis

Add make clean support to remove the containers from the local docker registry:
- make clean: remove all images
- clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm image

---
Changes in v2:
- none
Changes in v1:
- patch added

---
 automation/build/Makefile | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
automation/build/Makefile +16 -1
index b0dc68f00b70..9fbb49c8717e 100644
@@ -4,6 +4,17 @@ REGISTRY := registry.gitlab.com/xen-project/xen
 CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
 DOCKER_CMD ?= docker
 
+define CLEAN_RULE
+.PHONY: clean-$(1)
+clean-$(1):
+ifneq ($$(shell docker image ls -q $(REGISTRY)/$(subst /,:,$(1))),)
+	docker image rm $(REGISTRY)/$(subst /,:,$(1))
+endif
+
+clean: clean-$(1)
+
+endef
+
 help:
 	@echo "Builds containers for building Xen based on different distros"
 	@echo "To build one run 'make DISTRO/VERSION'. Available containers:"
@@ -18,5 +29,9 @@ help:
 
 $(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone
 
-.PHONY: all
+.PHONY: all clean
 all: $(CONTAINERS)
+
+clean:
+
+$(eval $(foreach img,$(CONTAINERS),$(call CLEAN_RULE,$(img))))
Signature · 1 line
2.25.1
#4 lore raw
In reply to Bertrand Marquis
Quoted 27 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The third patch is adding a way to easily clean locally created containers.

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

Quoted 2 lines

Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.

The build still doesn't work for me. I found the reason:

create archive failed: cpio: write failed - Cannot allocate memory

It is a "silly" out of memory error. I tried to solve the problem by adding:

export RPM_BUILD_NCPUS=8

at the beginning of build-yocto.sh but it didn't work. I realize that this error might be considered a workstation configuration error at my end but I cannot find a way past it. Any suggestions?

#5 lore raw

Hi Stefano,

Quoted 37 lines · Stefano Stabellini
Quoted 27 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The third patch is adding a way to easily clean locally created containers.

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

Quoted 2 lines

Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.

The build still doesn't work for me. I found the reason:

create archive failed: cpio: write failed - Cannot allocate memory

Quoted 8 lines

It is a "silly" out of memory error. I tried to solve the problem by adding:

export RPM_BUILD_NCPUS=8

at the beginning of build-yocto.sh but it didn't work. I realize that this error might be considered a workstation configuration error at my end but I cannot find a way past it. Any suggestions?

Can you give me more details on when this is happening ? Ie the full logs.

Can you try to apply the following:

automation/build/yocto/build-yocto.sh +3 -0
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space
 INHERIT += "rm_work"
 
+# Reduce number of jobs
+BB_NUMBER_THREADS=2
+
 EOF
 
     if [ "${do_localsrc}" = "y" ]; then

This should reduce the number of parallel jobs during Yocto build.

Cheers
Bertrand

#6 lore raw
Quoted 65 lines · Bertrand Marquis

Hi Stefano,

Quoted 37 lines · Stefano Stabellini
Quoted 27 lines · Bertrand Marquis

This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.

The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.

The third patch is adding a way to easily clean locally created containers.

This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab

Quoted 2 lines

Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.

The build still doesn't work for me. I found the reason:

create archive failed: cpio: write failed - Cannot allocate memory

Quoted 8 lines

It is a "silly" out of memory error. I tried to solve the problem by adding:

export RPM_BUILD_NCPUS=8

at the beginning of build-yocto.sh but it didn't work. I realize that this error might be considered a workstation configuration error at my end but I cannot find a way past it. Any suggestions?

Can you give me more details on when this is happening ? Ie the full logs.

Can you try to apply the following:

automation/build/yocto/build-yocto.sh +3 -0
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space
 INHERIT += "rm_work"
 
+# Reduce number of jobs
+BB_NUMBER_THREADS=2
+
 EOF
 
     if [ "${do_localsrc}" = "y" ]; then
Quoted 1 line

This should reduce the number of parallel jobs during Yocto build.

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

#7 lore raw
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193051236
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193083119

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +3 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space
 INHERIT += "rm_work"
 
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+
 EOF
 
     if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 
 FROM $from_image
 
automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
              /home/$USER_NAME/xen && \
     chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need
 ARG yocto_version="kirkstone"
 RUN for rep in \
automation/gitlab-ci/build.yaml +16 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
     EXTRA_XEN_CONFIG: |
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+
 ## Test artifacts common
 
 .test-jobs-artifact-common:
#8 lore raw

Hi Stefano,

Quoted 7 lines · Stefano Stabellini
Quoted 4 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

Quoted 7 lines
Quoted 1 line

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193051236
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193083119

Awesome, this is quite fast :-)

Quoted 5 lines

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64

This should already be handle by the Makefile using PUSH or did you have to modify something ?

Quoted 2 lines

- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container

Ok

Quoted 3 lines

- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree

Ok

Quoted 2 lines

- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it

I will put this by default and leave a command line argument to have a solution to change this.

Quoted 2 lines

- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

I tested this when I made the patches and this works for arm64, arm32 and x86 targets on an arm64 machine so go for it.

Quoted 3 lines

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

Will do and I will also add a patch to create the build.yaml entries.

Quoted 2 lines

I am looking forward to it! Almost there!

Me to :-)

Thanks a lot for the testing and the review.

Cheers
Bertrand

Quoted 75 lines

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +0 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space

INHERIT += "rm_work"

automation/build/yocto/build-yocto.sh +3 -0
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+

EOF

    if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 

FROM $from_image

automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
             /home/$USER_NAME/xen && \
    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need

ARG yocto_version="kirkstone"
RUN for rep in \

automation/gitlab-ci/build.yaml +16 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
    EXTRA_XEN_CONFIG: |
      CONFIG_BOOT_TIME_CPUPOOLS=y
 
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+
 ## Test artifacts common
 

.test-jobs-artifact-common:

#9 lore raw
In reply to Stefano Stabellini

Hi Stefano,

Quoted 90 lines · Stefano Stabellini
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=oWrGVbloqkJoOxvvxTr55RbKVzd3YmS4iiLPyxDZCYY%3D&amp;reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BXf3ZB1bsVi8K%2BzEEe1Dhpg0GSohpseogff12GaK3Gw%3D&amp;reserved=0

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +3 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space
 INHERIT += "rm_work"
 
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+
 EOF
 
     if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 
 FROM $from_image
 
automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
              /home/$USER_NAME/xen && \
     chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need
 ARG yocto_version="kirkstone"
 RUN for rep in \
automation/gitlab-ci/build.yaml +1 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
     EXTRA_XEN_CONFIG: |
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
+yocto-kirkstone-qemuarm64:

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 597

Quoted 10 lines
automation/gitlab-ci/build.yaml +10 -0
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'

The above lines are not needed as the logs/* below will handle them all (logs are only stored in logs/).

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 602

Quoted 5 lines
automation/gitlab-ci/build.yaml +5 -0
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+

build-yocto.sh performs both build and run actions. I think it'd be better to move this into test.yaml in that case. The best would be to create one build job (specifying --no-run) in build.yaml and one test job (specifying --no-build) in test.yaml. This however would probably require marking path build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is whether having this path would be enough for runqemu (Bertrand's opinion needed).

Apart from that there is an aspect of Yocto releases and the containers/tests names. Yocto needs to be up-to-date in order to properly build Xen+tools.
This basically means that we will need to update the containers once
per Yocto release. The old containers would still need to be stored in our CI container registry so that we can use CI for older versions of Xen. However, updating the containers would also require modifying the existing tests (for now we have e.g. yocto-kirkstone-qemuarm64 but in a month we will have to change them to yocto-langdale-qemuarm64). In a few years time this will result in several CI jobs that are the same but differ only in name/container. I would thus suggest to name the CI jobs like this:
yocto-qemuarm64 (without yocto release name) and define the top-level YOCTO_CONTAINER variable to store the current yocto release container. This will solve the issue I described above.

~Michal

#10 lore raw

Hi Michal,

Quoted 94 lines · Michal Orzel

Hi Stefano,

Quoted 89 lines · Stefano Stabellini
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=oWrGVbloqkJoOxvvxTr55RbKVzd3YmS4iiLPyxDZCYY%3D&amp;reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BXf3ZB1bsVi8K%2BzEEe1Dhpg0GSohpseogff12GaK3Gw%3D&amp;reserved=0

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +0 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space

INHERIT += "rm_work"

automation/build/yocto/build-yocto.sh +3 -0
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+

EOF

    if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 

FROM $from_image

automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
             /home/$USER_NAME/xen && \
    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need

ARG yocto_version="kirkstone"
RUN for rep in \

automation/gitlab-ci/build.yaml +0 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
    EXTRA_XEN_CONFIG: |
      CONFIG_BOOT_TIME_CPUPOOLS=y
 
Quoted 12 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 597

Quoted 11 lines
automation/gitlab-ci/build.yaml +11 -0
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'

The above lines are not needed as the logs/* below will handle them all (logs are only stored in logs/).

Ack

Quoted 10 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 602

Quoted 6 lines
automation/gitlab-ci/build.yaml +5 -0
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+

build-yocto.sh performs both build and run actions. I think it'd be better to move this into test.yaml in that case. The best would be to create one build job (specifying --no-run) in build.yaml and one test job (specifying --no-build) in test.yaml. This however would probably require marking path build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is whether having this path would be enough for runqemu (Bertrand's opinion needed).

This will not be enough to run qemu as the qemu binary and its dependencies are in the build artifacts and not in deploy. Splitting the build and run is not a good idea because the size of the artifact between the 2 will be huge.

Quoted 11 lines

Apart from that there is an aspect of Yocto releases and the containers/tests names. Yocto needs to be up-to-date in order to properly build Xen+tools.
This basically means that we will need to update the containers once
per Yocto release. The old containers would still need to be stored in our CI container registry so that we can use CI for older versions of Xen. However, updating the containers would also require modifying the existing tests (for now we have e.g. yocto-kirkstone-qemuarm64 but in a month we will have to change them to yocto-langdale-qemuarm64). In a few years time this will result in several CI jobs that are the same but differ only in name/container. I would thus suggest to name the CI jobs like this:
yocto-qemuarm64 (without yocto release name) and define the top-level YOCTO_CONTAINER variable to store the current yocto release container. This will solve the issue I described above.

I think we have no other way around this and we will need to have one Yocto release supported by Xen officially so we will have to keep old docker images for old releases of Xen and move to newer versions of Yocto in staging when it is needed.

We have to find a way for gitlab-ci to use the build.yaml contained inside the tree that is to be tested somehow so that gitlab would automatically take the right one. Which means that build.yaml will be different between branches and contain the right version for the current branch.

Regards
Bertrand

Quoted 3 lines

~Michal

#11 lore raw

Hi Bertrand,

Quoted 139 lines · Bertrand Marquis

Hi Michal,

Quoted 94 lines · Michal Orzel

Hi Stefano,

Quoted 89 lines · Stefano Stabellini
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=2mb3N26wiz39RJNSA4KoIOt%2BG9X7EMDOWIpfKc2ZZOc%3D&amp;reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=QhTFefS8NU1f7oLemB0Vtn%2BDCD%2BCnq1v1gEmlKCJt84%3D&amp;reserved=0

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +0 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space

INHERIT += "rm_work"

automation/build/yocto/build-yocto.sh +3 -0
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+

EOF

    if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 

FROM $from_image

automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
             /home/$USER_NAME/xen && \
    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need

ARG yocto_version="kirkstone"
RUN for rep in \

automation/gitlab-ci/build.yaml +0 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
    EXTRA_XEN_CONFIG: |
      CONFIG_BOOT_TIME_CPUPOOLS=y
 
Quoted 12 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 597

Quoted 11 lines
automation/gitlab-ci/build.yaml +11 -0
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'

The above lines are not needed as the logs/* below will handle them all (logs are only stored in logs/).

Ack

Quoted 10 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 602

Quoted 6 lines
automation/gitlab-ci/build.yaml +5 -0
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+

build-yocto.sh performs both build and run actions. I think it'd be better to move this into test.yaml in that case. The best would be to create one build job (specifying --no-run) in build.yaml and one test job (specifying --no-build) in test.yaml. This however would probably require marking path build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is whether having this path would be enough for runqemu (Bertrand's opinion needed).

This will not be enough to run qemu as the qemu binary and its dependencies are in the build artifacts and not in deploy. Splitting the build and run is not a good idea because the size of the artifact between the 2 will be huge.

Quoted 11 lines

Apart from that there is an aspect of Yocto releases and the containers/tests names. Yocto needs to be up-to-date in order to properly build Xen+tools.
This basically means that we will need to update the containers once
per Yocto release. The old containers would still need to be stored in our CI container registry so that we can use CI for older versions of Xen. However, updating the containers would also require modifying the existing tests (for now we have e.g. yocto-kirkstone-qemuarm64 but in a month we will have to change them to yocto-langdale-qemuarm64). In a few years time this will result in several CI jobs that are the same but differ only in name/container. I would thus suggest to name the CI jobs like this:
yocto-qemuarm64 (without yocto release name) and define the top-level YOCTO_CONTAINER variable to store the current yocto release container. This will solve the issue I described above.

Quoted 7 lines

I think we have no other way around this and we will need to have one Yocto release supported by Xen officially so we will have to keep old docker images for old releases of Xen and move to newer versions of Yocto in staging when it is needed.

We have to find a way for gitlab-ci to use the build.yaml contained inside the tree that is to be tested somehow so that gitlab would automatically take the right one. Which means that build.yaml will be different between branches and contain the right version for the current branch.

What I suggest is that with each new yocto release, we add new docker container files and push them to registry. So we will end up in a registry having e.g. (arm64 as an example):
- kirkstone-qemuarm64
- langdale-qemuarm64
We maintain only the one group of CI jobs whose names are generic (yocto-qemuarm64).
After adding new containers for a new Yocto release, we modify the YOCTO_RELEASE variable
to point to the latest yocto release containers.

test.yaml:
...
# Yocto test jobs
variables:
YOCTO_RELEASE: "kirkstone"

yocto-qemuarm64:
extends: .test-jobs-common
script:
- ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=pwd qemuarm64
variables:

    CONTAINER: yocto:${YOCTO_RELEASE}-qemuarm64

artifacts:
paths:
- 'logs/*'
when: always
tags:
- arm64

This means that:
- on the current staging branch the YOCTO_RELEASE points to the latest containers (for the latest yocto release)
- on the old stable branches the YOCTO_RELEASE points to the old containers (for the old yocto release).

~Michal

#12 lore raw
Quoted 177 lines · Michal Orzel

Hi Bertrand,

Quoted 139 lines · Bertrand Marquis

Hi Michal,

Quoted 94 lines · Michal Orzel

Hi Stefano,

Quoted 89 lines · Stefano Stabellini
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=2mb3N26wiz39RJNSA4KoIOt%2BG9X7EMDOWIpfKc2ZZOc%3D&amp;reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=QhTFefS8NU1f7oLemB0Vtn%2BDCD%2BCnq1v1gEmlKCJt84%3D&amp;reserved=0

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +0 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space

INHERIT += "rm_work"

automation/build/yocto/build-yocto.sh +3 -0
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+

EOF

    if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 

FROM $from_image

automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
             /home/$USER_NAME/xen && \
    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need

ARG yocto_version="kirkstone"
RUN for rep in \

automation/gitlab-ci/build.yaml +0 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
    EXTRA_XEN_CONFIG: |
      CONFIG_BOOT_TIME_CPUPOOLS=y
 
Quoted 12 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 597

Quoted 11 lines
automation/gitlab-ci/build.yaml +11 -0
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'

The above lines are not needed as the logs/* below will handle them all (logs are only stored in logs/).

Ack

Quoted 10 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 602

Quoted 6 lines
automation/gitlab-ci/build.yaml +5 -0
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+

build-yocto.sh performs both build and run actions. I think it'd be better to move this into test.yaml in that case. The best would be to create one build job (specifying --no-run) in build.yaml and one test job (specifying --no-build) in test.yaml. This however would probably require marking path build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is whether having this path would be enough for runqemu (Bertrand's opinion needed).

This will not be enough to run qemu as the qemu binary and its dependencies are in the build artifacts and not in deploy. Splitting the build and run is not a good idea because the size of the artifact between the 2 will be huge.

Quoted 11 lines

Apart from that there is an aspect of Yocto releases and the containers/tests names. Yocto needs to be up-to-date in order to properly build Xen+tools.
This basically means that we will need to update the containers once
per Yocto release. The old containers would still need to be stored in our CI container registry so that we can use CI for older versions of Xen. However, updating the containers would also require modifying the existing tests (for now we have e.g. yocto-kirkstone-qemuarm64 but in a month we will have to change them to yocto-langdale-qemuarm64). In a few years time this will result in several CI jobs that are the same but differ only in name/container. I would thus suggest to name the CI jobs like this:
yocto-qemuarm64 (without yocto release name) and define the top-level YOCTO_CONTAINER variable to store the current yocto release container. This will solve the issue I described above.

Quoted 7 lines

I think we have no other way around this and we will need to have one Yocto release supported by Xen officially so we will have to keep old docker images for old releases of Xen and move to newer versions of Yocto in staging when it is needed.

We have to find a way for gitlab-ci to use the build.yaml contained inside the tree that is to be tested somehow so that gitlab would automatically take the right one. Which means that build.yaml will be different between branches and contain the right version for the current branch.

What I suggest is that with each new yocto release, we add new docker container files and push them to registry. So we will end up in a registry having e.g. (arm64 as an example):
- kirkstone-qemuarm64
- langdale-qemuarm64
We maintain only the one group of CI jobs whose names are generic (yocto-qemuarm64).
After adding new containers for a new Yocto release, we modify the YOCTO_RELEASE variable
to point to the latest yocto release containers.

test.yaml:
...
# Yocto test jobs
variables:
YOCTO_RELEASE: "kirkstone"

yocto-qemuarm64:
extends: .test-jobs-common
script:
- ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=pwd qemuarm64
variables:

    CONTAINER: yocto:${YOCTO_RELEASE}-qemuarm64

artifacts:
paths:
- 'logs/*'
when: always
tags:
- arm64

Quoted 3 lines

This means that:
- on the current staging branch the YOCTO_RELEASE points to the latest containers (for the latest yocto release)
- on the old stable branches the YOCTO_RELEASE points to the old containers (for the old yocto release).

I think that's a good idea

#13 lore raw
In reply to Bertrand Marquis
Quoted 117 lines · Bertrand Marquis

Hi Michal,

Quoted 94 lines · Michal Orzel

Hi Stefano,

Quoted 89 lines · Stefano Stabellini
Quoted 5 lines · Stefano Stabellini

It should be

BB_NUMBER_THREADS="2"

but that worked! Let me a couple of more tests.

I could run successfully a Yocto build test with qemuarm64 as target in gitlab-ci, hurray! No size issues, no build time issues, everything was fine. See:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=oWrGVbloqkJoOxvvxTr55RbKVzd3YmS4iiLPyxDZCYY%3D&amp;reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C75ea919bbde249e1bac408dab1654960%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017345841386870%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2BXf3ZB1bsVi8K%2BzEEe1Dhpg0GSohpseogff12GaK3Gw%3D&amp;reserved=0

I made the appended changes in top of this series.

- I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
- for the gitlab-ci runs, we need to run build-yocto.sh from the copy in xen.git, not from a copy stored inside a container
- when building the kirkstone-qemuarm64 container the first time (outside of gitlab-ci) I used COPY and took the script from the local xen.git tree
- after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than this and it breaks on some workstations, please add it
- I am running the yocto build on arm64 so that we can use the arm64 hardware to do it in gitlab-ci

Please feel free to incorporate these changes in your series, and add corresponding changes for the qemuarm32 and qemux86 targets.

I am looking forward to it! Almost there!

Cheers,

Stefano

automation/build/yocto/build-yocto.sh +0 -0
index 0d31dad607..16f1dcc0a5 100755
@@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
 # Save some disk space

INHERIT += "rm_work"

automation/build/yocto/build-yocto.sh +3 -0
+# Reduce number of jobs
+BB_NUMBER_THREADS="8"
+

EOF

    if [ "${do_localsrc}" = "y" ]; then
automation/build/yocto/kirkstone-qemuarm64.dockerfile +2 -1
index f279a7af92..aea3fc1f3e 100644
@@ -16,7 +16,8 @@ ARG target=qemuarm64
 
 # This step can take one to several hours depending on your download bandwith
 # and the speed of your computer
-RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log $target
 

FROM $from_image

automation/build/yocto/kirkstone.dockerfile +0 -3
index 367a7863b6..ffbd91aa90 100644
@@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
             /home/$USER_NAME/xen && \
    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
 
-# Copy the build script
-COPY build-yocto.sh /home/$USER_NAME/bin/
-
 # clone yocto repositories we need

ARG yocto_version="kirkstone"
RUN for rep in \

automation/gitlab-ci/build.yaml +0 -0
index ddc2234faf..4b8bcde252 100644
@@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
    EXTRA_XEN_CONFIG: |
      CONFIG_BOOT_TIME_CPUPOOLS=y
 
Quoted 12 lines

automation/gitlab-ci/build.yaml alpine-3.12-gcc-arm64-boot-cpupools: · line 597

Quoted 11 lines
automation/gitlab-ci/build.yaml +11 -0
+yocto-kirkstone-qemuarm64:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` qemuarm64
+  variables:
+    CONTAINER: yocto:kirkstone-qemuarm64
+  artifacts:
+    paths:
+      - '*.log'
+      - '*/*.log'

The above lines are not needed as the logs/* below will handle them all (logs are only stored in logs/).

Ack

Quoted 6 lines
Quoted 6 lines
automation/gitlab-ci/build.yaml +5 -0
+      - 'logs/*'
+    when: always
+  tags:
+    - arm64
+
Quoted 7 lines
Quoted 4 lines

build-yocto.sh performs both build and run actions. I think it'd be better to move this into test.yaml in that case. The best would be to create one build job (specifying --no-run) in build.yaml and one test job (specifying --no-build) in test.yaml. This however would probably require marking path build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is whether having this path would be enough for runqemu (Bertrand's opinion needed).

This will not be enough to run qemu as the qemu binary and its dependencies are in the build artifacts and not in deploy. Splitting the build and run is not a good idea because the size of the artifact between the 2 will be huge.

Although not ideal, I think it is fine to have a single job that does both the build and the run.