Yocto Gitlab CI
v3 cover Yocto Gitlab CI
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab
Once all problems passed, this can be used to build and run dom0 on qemu with a modified Xen on the 3 archs in less than 10 minutes.
This has been tested on a x86 host machine and on an arm host machine (with mk_dsdt.c fix).
Changes in v3:
- limit number of jobs in yocto
- do not copy build script inside container
- add patch from Michal to create gitlab jobs
Changes in v2:
- remove gitignore patch which was merged
- add a --dump-log support in build-yocto.sh script and use it during container creation to see the error logs.
Bertrand Marquis (3):
automation: Add elements for Yocto test and run
automation: Add yocto containers with cache
automation: Add a clean rule for containers
Michal Orzel (1):
automation: Add CI test jobs for Yocto
automation/build/Makefile | 19 +- automation/build/yocto/build-yocto.sh | 349 ++++++++++++++++++ .../build/yocto/kirkstone-qemuarm.dockerfile | 29 ++ .../yocto/kirkstone-qemuarm64.dockerfile | 29 ++ .../yocto/kirkstone-qemux86-64.dockerfile | 29 ++ automation/build/yocto/kirkstone.dockerfile | 97 +++++ automation/gitlab-ci/test.yaml | 34 ++ 7 files changed, 585 insertions(+), 1 deletion(-) create mode 100755 automation/build/yocto/build-yocto.sh create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile create mode 100644 automation/build/yocto/kirkstone.dockerfile
Signature · 1 line
2.25.1
PATCH v3 1/4 automation: Add elements for Yocto test and run
Add a container suitable to run yocto kirkstone build based on ubuntu 22.04. It contains all packages required by Yocto and a checkout of the layers required to build Xen with Yocto.
Add a build script to build and run xen on qemu using Yocto. The script supports arm32, arm64 and x86_64 and checks that dom0 is properly booting. At this stage this does not run any guest on top of dom0.
The build script is currently in the same directory than the docker files so that it can be included directly in the image. This is required for the next step which creates images with an initial cache content to speed up testing.
---
Changes in v3:
- limit number of jobs in yocto by default to 8 and add --num-jobs option to the script to set a custom number of jobs
- do not copy anymore the build-yocto.sh script inside the main image so that the current one in the repository is used when running Changes in v2:
- add a --dump-log command line option to build-yocto.sh script to dump the logs if an error occurs.
Changes in v1:
- add --image command line argument to build-yocto.sh to allow building something different than xen-image-minimal.
- modify dockerfile to have one layer per line and make it easier to add other. I kept the for loop to keep the number of docker steps lower
- update commit message to warn that no guest are tested.
- fix build-yocto script to properly return with an error if expect script ends up in timeout or EOF.
--- automation/build/yocto/build-yocto.sh | 349 ++++++++++++++++++++ automation/build/yocto/kirkstone.dockerfile | 97 ++++++ 2 files changed, 446 insertions(+) create mode 100755 automation/build/yocto/build-yocto.sh create mode 100644 automation/build/yocto/kirkstone.dockerfile
automation/build/yocto/build-yocto.sh new file +349 -0
@@ -0,0 +1,349 @@ +#!/bin/bash +# +# Yocto meta virtualization build and run script +# +# This script is building Yocto xen-image-minimal for qemu targets and run +# them using runqemu inside yocto to check that dom0 is booting properly +# The build is using a local xen source tree so that specific patches can be +# tested. +# In order to optimize the build time, a build cache is used so that only xen +# packages and its dependencies are rebuilt (qemu and final image mainly). +# + +# Directories +YOCTODIR="$HOME/yocto-layers" +CACHEDIR="$HOME/yocto-cache" +LOGDIR="$HOME/logs" +XENDIR="$HOME/xen" +BUILDDIR="$HOME/build" + +# what yocto bsp we support +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64" +VERBOSE="n" +TARGETLIST="" +BUILDJOBS="8" + +# actions to do +do_clean="n" +do_build="y" +do_run="y" +do_localsrc="n" +do_dump="n" +build_result=0 + +# layers to include in the project +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \ + meta-openembedded/meta-oe meta-openembedded/meta-python \ + meta-openembedded/meta-filesystems \ + meta-openembedded/meta-networking meta-virtualization" + +# yocto image to build +build_image="xen-image-minimal" + +function print_progress() { + echo -n "$(date +%T) $*" +} + +function run_task() { + local task_name="$1" + local task_target="$2" + + task_log="${task_name//project_}-${task_target}" + + mkdir -p "${LOGDIR}" + print_progress + echo -n "${task_name//project_} ${task_target}: " + if [ "${VERBOSE}" = "n" ]; then + $@ > "${LOGDIR}/${task_log}.log" 2>&1 + else + $@ 2>&1 | tee "${LOGDIR}/${task_log}.log" + fi + + if [ ${?} -ne 0 ]; then + echo "Error" + build_result=$((build_result+1)) + if [ "${do_dump}" = "y" ]; then + echo + echo "############ LOGS-START ############" + cat "${LOGDIR}/${task_log}.log" + echo "############ LOGS-END ############" + echo + fi + return 1 + else + echo "OK" + return 0 + fi +} + +function project_create() { + target="${1:?}" + destdir="${BUILDDIR}/${target}" + + ( + # init yocto project + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" + + # add needed layers + for layer in ${build_layerlist}; do + bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1 + done + ) || return 1 + + # customize project configuration + cat <<EOF >> "${destdir}/conf/local.conf" +# Yocto BSP +MACHINE = "${target}" + +# Use local cache to reuse previous builds results +SSTATE_DIR = "${CACHEDIR}/sstate-cache" +DL_DIR = "${CACHEDIR}/downloads" + +# Enable xen and virtualization +DISTRO_FEATURES = " virtualization xen ipv4" + +# Speed up run by not generating ssh host keys +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys" + +# Save some disk space +INHERIT += "rm_work" + +# Reduce number of jobs +BB_NUMBER_THREADS="${BUILDJOBS}" + +EOF + + if [ "${do_localsrc}" = "y" ]; then + XENVERS=$(grep -e "^XEN_REL" \ + "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \ + 2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \ + | sed -e "s/.*=//" | sort -V | tail -n 1) + + XENBASE=$(dirname "$(realpath -m "${XENDIR}")") + XENSUB=$(basename "$(realpath -m "${XENDIR}")") + + cat <<EOF >> "${destdir}/conf/local.conf" +# Use local sources for xen and xen-tools +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:" +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:" + +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/" +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/" + +PREFERRED_VERSION:pn-xen = "${XENVERS}%" +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%" + +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}" +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}" + +SRCREV:pn-xen = "\${AUTOREV}" +SRCREV:pn-xen-tools = "\${AUTOREV}" + +SRCPV:pn-xen = "1" +SRCPV:pn-xen-tools = "1" + +# Disable all QA errors as the recipe is not up to date with changes in Xen +# when we use local sources +ERROR_QA:pn-xen = "arch" +ERROR_QA:pn-xen-tools = "arch" + +EOF + fi +} + +function project_build() { + target="${1:?}" + destdir="${BUILDDIR}/${target}" + + ( + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" + + bitbake "${build_image}" || exit 1 + ) || return 1 +} + +function project_clean() { + target="${1:?}" + destdir="${BUILDDIR}/${target}" + + rm -rf "${destdir}" +} + +function project_run() { + target="${1:?}" + destdir="${BUILDDIR}/${target}" + ( + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1 + + /usr/bin/expect <<EOF +set timeout 100 +spawn bash -c "runqemu serialstdio nographic slirp" + +expect_after { + -re "(.*)\r" { + exp_continue + } + timeout {send_user "ERROR-Timeout!\n"; exit 1} + eof {send_user "ERROR-EOF!\n"; exit 1} +} + +# wait initial login +expect -re ".* login: " +send "root\r" +expect -re "root@.*# " + +EOF + exit $? + ) || return 1 +} + +function help() { + cat <<EOF +Usage: ${0} [TARGET1] [TARGET2] + +This script is build the yocto xen-image-minimal for different qemu targets +and is running it after. +Without any target specified, all supported targets are done. + +Options: + -h, --help Print this help + -v, --verbose Verbose build + --list-target List supported targets + --clean Clean existing project before starting + --no-build Do not build (to run an already built project) + --no-run Do not run + --num-jobs=NUM Define the number of parallel jobs in Yocto. + Default: ${BUILDJOBS} + --dump-log On error, dump the logs on the console + --image=IMG Yocto image or package to build + Default: xen-image-minimal + --xen-dir=DIR path to xen hypervisor source tree + if not provide, normal yocto version of xen is built + Default: ${XENDIR} + --out-dir=DIR directory where to create the projectss + Default: ${BUILDDIR} + --log-dir=DIR directory to store logs + Default: ${LOGDIR} + --cache-dir=DIR directory where to take and store build cache + Default: ${CACHEDIR} + --layer-dir=DIR directory containing the checkout of yocto layers + Default: ${YOCTODIR} +EOF +} + +for OPTION in "$@" +do + case ${OPTION} in + -h|--help) + help + exit 0 + ;; + -v|--verbose) + VERBOSE="y" + ;; + --list-targets) + echo "${TARGET_SUPPORTED}" + exit 0 + ;; + --clean) + do_clean="y" + ;; + --no-build) + do_build="n" + ;; + --no-run) + do_run="n" + ;; + --dump-log) + do_dump="y" + ;; + --num-jobs=*) + BUILDJOBS="${OPTION#*=}" + ;; + --image=*) + build_image="${OPTION#*=}" + ;; + --xen-dir=*) + XENDIR="${OPTION#*=}" + if [ ! -e "${XENDIR}/xen/Makefile" ]; then + echo "No Xen source tree in ${XENDIR}" + exit 1 + fi + do_localsrc="y" + ;; + --out-dir=*) + BUILDDIR="${OPTION#*=}" + ;; + --log-dir=*) + LOGDIR="${OPTION#*=}" + ;; + --cache-dir=*) + CACHEDIR="${OPTION#*=}" + ;; + --layer-dir=*) + YOCTODIR="${OPTION#*=}" + ;; + --*) + echo "Invalid option ${OPTION}" + help + exit 1 + ;; + *) + if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then + TARGETLIST="${TARGETLIST} ${OPTION}" + else + echo "Unsupported target ${OPTION}" + exit 1 + fi + ;; + esac +done + +# if no target is specified build all targets +if [ -z "${TARGETLIST}" ]; then + TARGETLIST="${TARGET_SUPPORTED}" +fi + +mkdir -p "${CACHEDIR}" +mkdir -p "${LOGDIR}" +mkdir -p "${BUILDDIR}" + +# Make sure we have an absolute path +YOCTODIR=$(realpath -m "${YOCTODIR}") +CACHEDIR=$(realpath -m "${CACHEDIR}") +BUILDDIR=$(realpath -m "${BUILDDIR}") +LOGDIR=$(realpath -m "${LOGDIR}") +if [ "${do_localsrc}" = "y" ]; then + XENDIR=$(realpath -m "${XENDIR}") +fi + +# Check that we have all the layers we need +for f in ${build_layerlist}; do + if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then + echo "Layer ${f} missing in ${YOCTODIR}" + exit 1 + fi +done + +for f in ${TARGETLIST}; do + if [ "${do_clean}" = "y" ]; then + run_task project_clean ${f} + fi + if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then + run_task project_create ${f} + fi + if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then + if [ "${do_build}" = "y" ]; then + run_task project_build ${f} + fi + if [ "${do_run}" = "y" ]; then + run_task project_run ${f} + fi + + fi +done + +print_progress "Build Complete (${build_result} errors)" +echo +exit ${build_result} +
automation/build/yocto/kirkstone.dockerfile new file +97 -0
97 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 2/4 automation: Add yocto containers with cache
In reply to Bertrand MarquisCreate yocto containers for kirkstone with a builder step to fill the yocto cache by running the build script once.
Those containers depend on the kirkstone.dockerfile container to be created and use it as base image.
To run a build test on one image the following command must be called inside the container:build-yocto.sh --xen-dir=XENSRC TARGET where
- XENSRC is the directory where xen sources are available in the container
- TARGET is the build target (qemuarm64, qemuarm or qemux86_64)
In case of error, the script will return an error code and the logs will be in /home/docker-build/logs.
!!!
The build of each container can take several hours depending on the network bandwith (for downloading all sources) and the amount of computing power on the machine where the docker images are created. !!!
---
Changes in v3:
- Copy build-yocto.sh inside the container before running it to generate the cache.
Changes in v2:
- pass --dump-log option to build-yocto.sh during the image generation so that we have more info when there is an error during the image creation.
Changes in v1:
- fix dependency in Makefile
--- automation/build/Makefile | 2 ++ .../build/yocto/kirkstone-qemuarm.dockerfile | 29 +++++++++++++++++++ .../yocto/kirkstone-qemuarm64.dockerfile | 29 +++++++++++++++++++ .../yocto/kirkstone-qemux86-64.dockerfile | 29 +++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
automation/build/Makefile +2 -0
@@ -16,5 +16,7 @@ help: $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \ fi +$(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone + .PHONY: all all: $(CONTAINERS)
automation/build/yocto/kirkstone-qemuarm.dockerfile new file +29 -0
@@ -0,0 +1,29 @@ +# Docker file to populate the initial sstate and download cache for yocto +# for a specific target +# +# This requires the build-yocto.sh script to be in the same directory as this +# dockerfile during the image creation +# +# Arguments that can be passed during image creation using --build-arg: +# "from_image=img" to select the docker image and tag to start from +# "target=qemuxxx": to select which target to build in this image + +ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone + +FROM $from_image AS builder + +ARG target=qemuarm + +# This step can take one to several hours depending on your download bandwith +# and the speed of your computer +COPY ./build-yocto.sh / +RUN /build-yocto.sh --dump-log $target + +FROM $from_image + +# Only copy the cache status +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/. + +LABEL maintainer.name="The Xen Project " \ + maintainer.email="[email protected]" +
automation/build/yocto/kirkstone-qemuarm64.dockerfile new file +29 -0
@@ -0,0 +1,29 @@ +# Docker file to populate the initial sstate and download cache for yocto +# for a specific target +# +# This requires the build-yocto.sh script to be in the same directory as this +# dockerfile during the image creation +# +# Arguments that can be passed during image creation using --build-arg: +# "from_image=img" to select the docker image and tag to start from +# "target=qemuxxx": to select which target to build in this image + +ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone + +FROM $from_image AS builder + +ARG target=qemuarm64 + +# This step can take one to several hours depending on your download bandwith +# and the speed of your computer +COPY ./build-yocto.sh / +RUN /build-yocto.sh --dump-log $target + +FROM $from_image + +# Only copy the cache status +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/. + +LABEL maintainer.name="The Xen Project " \ + maintainer.email="[email protected]" +
automation/build/yocto/kirkstone-qemux86-64.dockerfile new file +29 -0
@@ -0,0 +1,29 @@ +# Docker file to populate the initial sstate and download cache for yocto +# for a specific target +# +# This requires the build-yocto.sh script to be in the same directory as this +# dockerfile during the image creation +# +# Arguments that can be passed during image creation using --build-arg: +# "from_image=img" to select the docker image and tag to start from +# "target=qemuxxx": to select which target to build in this image + +ARG from_image=registry.gitlab.com/xen-project/xen/yocto:kirkstone + +FROM $from_image AS builder + +ARG target=qemux86-64 + +# This step can take one to several hours depending on your download bandwith +# and the speed of your computer +COPY ./build-yocto.sh / +RUN /build-yocto.sh --dump-log $target + +FROM $from_image + +# Only copy the cache status +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/. + +LABEL maintainer.name="The Xen Project " \ + maintainer.email="[email protected]" +
Signature · 1 line
2.25.1
PATCH v3 3/4 automation: Add a clean rule for containers
In reply to Bertrand MarquisAdd make clean support to remove the containers from the local docker registry:
- make clean: remove all images
- clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm image
---
Changes in v3:
- none
Changes in v2:
- none
Changes in v1:
- patch added
--- automation/build/Makefile | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
automation/build/Makefile +16 -1
@@ -4,6 +4,17 @@ REGISTRY := registry.gitlab.com/xen-project/xen CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile)) DOCKER_CMD ?= docker +define CLEAN_RULE +.PHONY: clean-$(1) +clean-$(1): +ifneq ($$(shell docker image ls -q $(REGISTRY)/$(subst /,:,$(1))),) + docker image rm $(REGISTRY)/$(subst /,:,$(1)) +endif + +clean: clean-$(1) + +endef + help: @echo "Builds containers for building Xen based on different distros" @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
@@ -18,5 +29,9 @@ help: $(subst .dockerfile,,$(wildcard yocto/kirkstone-*)): yocto/kirkstone -.PHONY: all +.PHONY: all clean all: $(CONTAINERS) + +clean: + +$(eval $(foreach img,$(CONTAINERS),$(call CLEAN_RULE,$(img))))
Signature · 1 line
2.25.1
PATCH v3 4/4 automation: Add CI test jobs for Yocto
In reply to Bertrand MarquisFrom: Michal Orzel <[email protected]>
Populate test jobs for Yocto based tests using the provided containers. Due to the size restrictions, it is currently not possible to split the build and run tasks, therefore everything is done in a single step.
Test jobs for the supported Yocto targets are generic to avoid the necessity to add new ones after each Yocto release. The only thing required to be changed after updating the containers is the variable YOCTO_VERSION stored in a .yocto-test template.
---
Changes in v3:
- patch added
--- automation/gitlab-ci/test.yaml | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
automation/gitlab-ci/test.yaml +34 -0
@@ -52,6 +52,18 @@ tags: - x86_64 +.yocto-test: + extends: .test-jobs-common + script: + - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD} + variables: + YOCTO_VERSION: kirkstone + CONTAINER: yocto:${YOCTO_VERSION}-${YOCTO_BOARD} + artifacts: + paths: + - 'logs/*' + when: always + # Test jobs build-each-commit-gcc: extends: .test-jobs-common
@@ -186,3 +198,25 @@ qemu-smoke-x86-64-clang-pvh: - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE} needs: - debian-unstable-clang-debug + +# Yocto test jobs +yocto-qemuarm64: + extends: .yocto-test + variables: + YOCTO_BOARD: qemuarm64 + tags: + - arm64 + +yocto-qemuarm: + extends: .yocto-test + variables: + YOCTO_BOARD: qemuarm + tags: + - arm32 + +yocto-qemux86-64: + extends: .yocto-test + variables: + YOCTO_BOARD: qemux86-64 + tags: + - x86_64
Signature · 1 line
2.25.1
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are created
This is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
I will do some testing and get back to you with results + review.
~Michal
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
Thanks
Cheers
Bertrand
Quoted 2 lines
~Michal
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlab
Quoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 and x86_64). This means that all the test jobs would need to specify x86_64 as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/686853939
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 5c620fefce59..52cccec6f904 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -65,6 +65,9 @@
paths:
- 'logs/*'
when: always
+ needs: []
+ tags:
+ - x86_64
# Test jobs
build-each-commit-gcc:
@@ -206,19 +209,13 @@ yocto-qemuarm64:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarm64
- tags:
- - arm64
yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarm
- tags:
- - arm32
yocto-qemux86-64:
extends: .yocto-test
variables:
YOCTO_BOARD: qemux86-64
- tags:
- - x86_64
```Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
So we have 3 solutions:
1) Build and run these containers for/on x86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local users
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
~Michal
Hi Michal,
Quoted 108 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/686853939Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-0@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobsbuild-each-commit-gcc:
automation/gitlab-ci/test.yaml +0
-2@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarmautomation/gitlab-ci/test.yaml +0
-2- tags: - - arm32yocto-qemux86-64:
extends: .yocto-test
variables:YOCTO_BOARD: qemux86-64automation/gitlab-ci/test.yaml +0
-2- tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local users
I might have a solution 4.
The containers we pushed to gitlab are publicly accessible ?
Shouldn’t we push both versions there and then document how someone can retrieve and use those containers instead of building them ?
This way we will do the push but a normal user would just retrieve the containers and use them so building time would have less impact and we could build both flavours of the containers.
Quoted 4 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
Agree.
Bertrand
Quoted 2 lines
~Michal
Hi Bertrand,
Quoted 112 lines · Bertrand Marquis
Hi Michal,
Quoted 108 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7C097e7825c8834c412e8008dac16526bf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638034937437190896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MNo1wYRgxehJ0SJOeGxNR6U%2Fd68zERPgYfLbpHya%2B2Y%3D&reserved=0Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-0@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobsbuild-each-commit-gcc:
automation/gitlab-ci/test.yaml +0
-2@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarmautomation/gitlab-ci/test.yaml +0
-2- tags: - - arm32yocto-qemux86-64:
extends: .yocto-test
variables:YOCTO_BOARD: qemux86-64automation/gitlab-ci/test.yaml +0
-2- tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 14 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersI might have a solution 4.
The containers we pushed to gitlab are publicly accessible ?
Yes, they are:
https://gitlab.com/xen-project/xen/container_registry/3539006
Quoted 2 lines
Shouldn’t we push both versions there and then document how someone can retrieve and use those containers instead of building them ?
WRT retrieving, we already have a section describing it under automation/build/README.md
I'm not sure about pushing both versions to registry, due to the following:
- occupying twice the same size (instead of 30GB per release we would have 60GB, imagine the size after a few years),
- necessity to build twice the same containers for each release (someone would have to have access to both x86 and arm64 machines to do the build),
- necessity to introduce different naming for containers (to differentiate the machine) where in fact, CI would always use one subset.
We should not worry about the local users, because once the containers are in registry (no matter the platform they were built for), downloading the containers will take less time than building them. We will not benefit anything from having containers for different platforms. The only problem is with building them on x86 for arm64 (other way around is not supported by qemu-user-static). But this is something that may be problematic only for person willing to push to registry, because as you said, the normal users can pull them instead of build them.
Quoted 14 lines
This way we will do the push but a normal user would just retrieve the containers and use them so building time would have less impact and we could build both flavours of the containers.
Quoted 4 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
Agree.
Bertrand
Quoted 2 lines
~Michal
~Michal
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/686853939
When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-6@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobs build-each-commit-gcc:@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64 yocto-qemuarm: extends: .yocto-test variables: YOCTO_BOARD: qemuarm - tags: - - arm32 yocto-qemux86-64: extends: .yocto-test variables: YOCTO_BOARD: qemux86-64 - tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local users
From a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
Quoted 3 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
That's right, but it should be possible to allow the Yocto containers to also build and run correctly locally on x86, right? The arm/x86 tag in test.yaml doesn't matter when running the containers locally anyway.
Hi Stefano,
Quoted 113 lines · Stefano Stabellini
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7C2449f063e67341c3b95a08dac2b112a5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638036363027707274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EwTJrW2vuwQIugKc7mnzG9NNbsYLP6tw5UODzBMmPEE%3D&reserved=0When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-6@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobs build-each-commit-gcc:@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64 yocto-qemuarm: extends: .yocto-test variables: YOCTO_BOARD: qemuarm - tags: - - arm32 yocto-qemux86-64: extends: .yocto-test variables: YOCTO_BOARD: qemux86-64 - tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 16 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersFrom a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
That is what I reckon too. Running the Yocto build/test on CI using x86 runner will always be slower.
So, if we go with this solution, then the following is needed:
1. Modify test jobs so that yocto-qemu{arm64/arm} uses arm64 tag to be taken by arm64 runner and use tag x86_64 for yocto-qemux86-64.
2. Come up with a solution to build the yocto containers automatically for the above platforms + possibility to specify the platform for local users. Right now, these containers are being always build for the host machine platform, so without doing tricks like adding --platform or prefix to image name, one cannot build the Yocto containers that would be ready to be pushed to registry. We need to have a clean solution without requiring user to do tricks.
The only drawback of this solution is that the person building the yocto-qemu{arm64/arm} container and willing to push it to registry, needs to have access to arm64 machine.
Quoted 8 lines
Quoted 4 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
That's right, but it should be possible to allow the Yocto containers to also build and run correctly locally on x86, right? The arm/x86 tag in
test.yamldoesn't matter when running the containers locally anyway.
~Michal
Hi Michal,
Quoted 141 lines · Michal Orzel
Hi Stefano,
Quoted 113 lines · Stefano Stabellini
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7C2449f063e67341c3b95a08dac2b112a5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638036363027707274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EwTJrW2vuwQIugKc7mnzG9NNbsYLP6tw5UODzBMmPEE%3D&reserved=0When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-0@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobsbuild-each-commit-gcc:
automation/gitlab-ci/test.yaml +0
-2@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarmautomation/gitlab-ci/test.yaml +0
-2- tags: - - arm32yocto-qemux86-64:
extends: .yocto-test
variables:YOCTO_BOARD: qemux86-64automation/gitlab-ci/test.yaml +0
-2- tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 16 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersFrom a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
That is what I reckon too. Running the Yocto build/test on CI using x86 runner will always be slower.
So, if we go with this solution, then the following is needed:
1. Modify test jobs so that yocto-qemu{arm64/arm} uses arm64 tag to be taken by arm64 runner and use tagx86_64for yocto-qemux86-64.
2. Come up with a solution to build the yocto containers automatically for the above platforms + possibility to specify the platform for local users. Right now, these containers are being always build for the host machine platform, so without doing tricks like adding --platform or prefix to image name, one cannot build the Yocto containers that would be ready to be pushed to registry. We need to have a clean solution without requiring user to do tricks.
Quoted 2 lines
The only drawback of this solution is that the person building the yocto-qemu{arm64/arm} container and willing to push it to registry, needs to have access to arm64 machine.
I think we need to find a solution working for both possibilities.
And we also need a solution so that one can have both kind of images so the host machine should be encoded in the container name somehow.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
That's right, but it should be possible to allow the Yocto containers to also build and run correctly locally on x86, right? The arm/x86 tag in
test.yamldoesn't matter when running the containers locally anyway.
All in all, test.yaml only matter for gitlab.
Maybe we could have it supporting both cases but only use one ?
Cheers
Bertrand
Quoted 2 lines
~Michal
Hello,
Quoted 161 lines · Bertrand Marquis
Hi Michal,
Quoted 141 lines · Michal Orzel
Hi Stefano,
Quoted 113 lines · Stefano Stabellini
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7Ccc0a420856c64224e78208dac3037b95%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638036716985754450%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ARpspaYQlf7gujC11cS4dnR275cCJ6tMRC2J0FyhFIM%3D&reserved=0When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-0@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobsbuild-each-commit-gcc:
automation/gitlab-ci/test.yaml +0
-2@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarmautomation/gitlab-ci/test.yaml +0
-2- tags: - - arm32yocto-qemux86-64:
extends: .yocto-test
variables:YOCTO_BOARD: qemux86-64automation/gitlab-ci/test.yaml +0
-2- tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 16 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersFrom a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
That is what I reckon too. Running the Yocto build/test on CI using x86 runner will always be slower.
So, if we go with this solution, then the following is needed:
1. Modify test jobs so that yocto-qemu{arm64/arm} uses arm64 tag to be taken by arm64 runner and use tagx86_64for yocto-qemux86-64.
2. Come up with a solution to build the yocto containers automatically for the above platforms + possibility to specify the platform for local users. Right now, these containers are being always build for the host machine platform, so without doing tricks like adding --platform or prefix to image name, one cannot build the Yocto containers that would be ready to be pushed to registry. We need to have a clean solution without requiring user to do tricks.Quoted 2 lines
The only drawback of this solution is that the person building the yocto-qemu{arm64/arm} container and willing to push it to registry, needs to have access to arm64 machine.
I think we need to find a solution working for both possibilities.
And we also need a solution so that one can have both kind of images so the host machine should be encoded in the container name somehow.Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Regardless of what we chose, we need to keep in mind that the biggest advantage to the Yocto build/run is that it allows/should allow local users to perform basic testing for all the Xen supported architectures. This is because everything happens in one place with one command.
That's right, but it should be possible to allow the Yocto containers to also build and run correctly locally on x86, right? The arm/x86 tag in
test.yamldoesn't matter when running the containers locally anyway.
Quoted 9 lines
All in all,
test.yamlonly matter for gitlab.
Maybe we could have it supporting both cases but only use one ?Cheers
BertrandQuoted 2 lines
~Michal
I think we are trying to over engineer this problem. We want to start using it asap and this is what we should care about. If we start going into details, like adding more and more code (e.g take into consideration the fact that qemu emulation works only on x86 and what about users having arm64 machine), we will start creating a mess.
CURRENT BEHAVIOR:
All the containers are build for the host machine architecture.
For me, it can stay as a default behavior so that all the local users can make use of it.
WHAT WE WANT:
The default behavior like above and the possibility to specify the platform the containers will be build for. This is so that we can build the containers for arm on x86 and use it for registry to be used by CI.
RESULT:
Default behavior is used by local users allowing to perform relatively quick build and test. Possibility to specify the platform is used by the person building and pushing the containers for CI runners.
We do not need any duplication in containers' naming or in test.yaml test jobs. Test jobs specify tags to link the job with a proper runner and that is why the registry should only have containers built for proper runners' architectures.
~Michal
Quoted 139 lines · Michal Orzel
Hi Stefano,
Quoted 113 lines · Stefano Stabellini
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7C2449f063e67341c3b95a08dac2b112a5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638036363027707274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EwTJrW2vuwQIugKc7mnzG9NNbsYLP6tw5UODzBMmPEE%3D&reserved=0When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-6@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobs build-each-commit-gcc:@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64 yocto-qemuarm: extends: .yocto-test variables: YOCTO_BOARD: qemuarm - tags: - - arm32 yocto-qemux86-64: extends: .yocto-test variables: YOCTO_BOARD: qemux86-64 - tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 16 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersFrom a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
That is what I reckon too. Running the Yocto build/test on CI using x86 runner will always be slower.
So, if we go with this solution, then the following is needed:
1. Modify test jobs so that yocto-qemu{arm64/arm} uses arm64 tag to be taken by arm64 runner and use tagx86_64for yocto-qemux86-64.
2. Come up with a solution to build the yocto containers automatically for the above platforms + possibility to specify the platform for local users. Right now, these containers are being always build for the host machine platform, so without doing tricks like adding --platform or prefix to image name, one cannot build the Yocto containers that would be ready to be pushed to registry. We need to have a clean solution without requiring user to do tricks.
Quoted 2 lines
The only drawback of this solution is that the person building the yocto-qemu{arm64/arm} container and willing to push it to registry, needs to have access to arm64 machine.
I am fine with this drawback for now.
Due to resource constraints, we might want to avoid adding yocto-qemux86-64 (if yocto-qemux86-64 has to run on x86) for now, I worry it might choke the x86 ci-loop. Or we could add it but keep it disabled. We'll enable it when we get better x86 runners.
Hi Stefano,
Quoted 145 lines · Stefano Stabellini
Quoted 139 lines · Michal Orzel
Hi Stefano,
Quoted 113 lines · Stefano Stabellini
Quoted 55 lines · Michal Orzel
Hi Bertrand and Stefano,
Quoted 41 lines · Bertrand Marquis
Hi Michal,
Quoted 37 lines · Michal Orzel
Hi Bertrand,
Quoted 13 lines · Bertrand Marquis
This patch series is a first attempt to check if we could use Yocto in gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is creating a container with all elements required to build Yocto, a checkout of the yocto layers required and an helper script to build and run xen on qemu with yocto.
The second patch is creating containers with a first build of yocto done so that susbsequent build with those containers would only rebuild what was changed and take the rest from the cache.
Quoted 8 lines
The third patch is adding a way to easily clean locally created containers.
This is is mainly for discussion and sharing as there are still some issues/problem to solve:
- building the qemu* containers can take several hours depending on the network bandwith and computing power of the machine where those are createdThis is not really an issue as the build of the containers occurs on the local machines before pushing them to registry. Also, building the containers will only be required for new Yocto releases.
Quoted 7 lines
- produced containers containing the cache have a size between 8 and 12GB depending on the architecture. We might need to store the build cache somewhere else to reduce the size. If we choose to have one single image, the needed size is around 20GB and we need up to 40GB during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is over the allowed size in gitlabQuoted 9 lines
Quoted 3 lines
As we could see during v2 testing, we do not have any space restrictions on the Xen GitLab and I think we already decided to have the Yocto integrated into our CI.
Right, I should have modified this chapter to be coherent with your latest tests. Sorry for that.
Quoted 2 lines
I will do some testing and get back to you with results + review.
I did some testing and here are the results:
Quoted 7 lines
In the current form this series will fail when running CI because the Yocto containers
are based on "From ubuntu:22.04" (there is no platform prefix), which means that the containers are built for the host architecture (in my case and in 99% of the cases of the local build it will be x86). In Gitlab we have 2 runners (arm64 andx86_64). This means that all the test jobs would need to specifyx86_64as a tag when keeping the current behavior.
After I built all the containers on my x86 machine, I pushed them to registry and the pipeline was successful:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fmorzel%2Fxen-orzelmichal%2F-%2Fpipelines%2F686853939&data=05%7C01%7Cmichal.orzel%40amd.com%7C2449f063e67341c3b95a08dac2b112a5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638036363027707274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EwTJrW2vuwQIugKc7mnzG9NNbsYLP6tw5UODzBMmPEE%3D&reserved=0When I tested the previous version of this series I built the containers natively on ARM64, so that is also an option.
Quoted 43 lines
Here is the diff on patch no. 3 to make the series work (using x86 tag and small improvement to include needs: []):
```automation/gitlab-ci/test.yaml +3
-0@@ -65,6 +65,9 @@ paths: - 'logs/*' when: always + needs: [] + tags: + - x86_64 # Test jobsbuild-each-commit-gcc:
automation/gitlab-ci/test.yaml +0
-2@@ -206,19 +209,13 @@ yocto-qemuarm64: extends: .yocto-test variables: YOCTO_BOARD: qemuarm64 - tags: - - arm64yocto-qemuarm:
extends: .yocto-test
variables:
YOCTO_BOARD: qemuarmautomation/gitlab-ci/test.yaml +0
-2- tags: - - arm32yocto-qemux86-64:
extends: .yocto-test
variables:YOCTO_BOARD: qemux86-64automation/gitlab-ci/test.yaml +0
-2- tags: - - x86_64```
Now, the logical way would be to build x86 yocto container for x86, arm64 for arm64 and arm32 on arm64 or x86. I tried building the container qemuarm64 specifying target arm64 on x86. After 15h, only 70% of the Yocto build was completed and there was an error with glibc (the local build of the container for the host arch takes on my machine max 2h). This enormous amount of time is due to the qemu docker emulation that happens behind the scenes (I checked on 2 different machines).
Quoted 16 lines
Quoted 11 lines
So we have 3 solutions:
1) Build and run these containers for/onx86_64:
- local users can build the containers on local machines that are almost always x86 based, in short period of time,
- "everyone" can build/push the containers once there is a new Yocto release
- slightly slower CI build time
2) Build and run these containers for specific architectures:
- almost no go for local users using x86 machine (unless using more than 16 threads (which I used) and willing to wait 2 days for the build)
- faster CI build time (arm64 runner is faster than x86 one)
- someone with arm64 based machine (not that common) would have to build and push the containers
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local usersFrom a gitlab-ci perspective, given the runners we currently have, we have to go with option 2). We don't have enough resources available on the x86 runner to run the Yocto jobs on x86.
That is what I reckon too. Running the Yocto build/test on CI using x86 runner will always be slower.
So, if we go with this solution, then the following is needed:
1. Modify test jobs so that yocto-qemu{arm64/arm} uses arm64 tag to be taken by arm64 runner and use tagx86_64for yocto-qemux86-64.
2. Come up with a solution to build the yocto containers automatically for the above platforms + possibility to specify the platform for local users. Right now, these containers are being always build for the host machine platform, so without doing tricks like adding --platform or prefix to image name, one cannot build the Yocto containers that would be ready to be pushed to registry. We need to have a clean solution without requiring user to do tricks.Quoted 2 lines
The only drawback of this solution is that the person building the yocto-qemu{arm64/arm} container and willing to push it to registry, needs to have access to arm64 machine.
Quoted 6 lines
I am fine with this drawback for now.
Due to resource constraints, we might want to avoid adding yocto-qemux86-64 (if yocto-qemux86-64 has to run on x86) for now, I worry it might choke the x86 ci-loop. Or we could add it but keep it disabled. We'll enable it when we get better x86 runners.
Yocto-qemux86 actually runs quite well on an arm64 platform :-)
Cheers
Bertrand
Quoted 2 lines · Michal Orzel
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local users
FYI, it's already possible to build and push container from the CI, at least on X86, I've made it work:
https://lore.kernel.org/all/[email protected]/
This works only when pushing to the "staging" branch on the main "xen" repo as I've added secret variable there. (Also the branch needs to be "protected" if I remember correctly.)
I don't know how to use the Arm runner to do container builds like that.
(The change done to one x86 runner:
https://gitlab.com/xen-project/xen-gitlab-ci/-/merge_requests/15)
Cheers,
Signature · 1 line
Anthony PERARD
Hi Anthony,
Quoted 6 lines · Anthony PERARD
Quoted 2 lines · Michal Orzel
3) Try to use CI to build and push the containers to registry
- it could be possible but what about local users
Quoted 6 lines
FYI, it's already possible to build and push container from the CI, at least on X86, I've made it work:
https://lore.kernel.org/all/[email protected]/This works only when pushing to the "staging" branch on the main "xen" repo as I've added secret variable there. (Also the branch needs to be "protected" if I remember correctly.)
Very nice :-)
Would definitely be a good solution to have a solution like that for yocto.
Quoted 5 lines
I don't know how to use the Arm runner to do container builds like that.
(The change done to one x86 runner:
https://gitlab.com/xen-project/xen-gitlab-ci/-/merge_requests/15)
Something to look at definitely.
Cheers
Bertrand
Quoted 5 lines
Cheers,
Signature · 1 line
Anthony PERARD