Skip to contentSkip to conversation

CI: Fixes/cleanup in preparation for RISCV

PATCH In review Andrew Cooper 4 participants 28 messages Dec 30, 2022 – Jan 4, 2023

lore
Share & tools
lore
#0 lore raw

cover CI: Fixes/cleanup in preparation for RISCV

This series is intended to avoid the need for any editing of automation/scripts/build in the main RISCV series.

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/734584203
https://gitlab.com/xen-project/people/andyhhp/xen/-/commits/auto-build

The two Ubuntu Bionic failures are known and preexisting.

Andrew Cooper (6):
CI: Drop automation/configs/
CI: Remove guesswork about which artefacts to preserve
CI: Only calculate ./configure args if needed
CI: Express HYPERVISOR_ONLY in build.yml
CI: Fix build script when CROSS_COMPILE is in use
CI: Simplify the MUSL check

 automation/configs/x86/hvm_only_config  |   3 -
 automation/configs/x86/no_hvm_pv_config |   3 -
 automation/configs/x86/pv_only_config   |   3 -
 automation/gitlab-ci/build.yaml         |   2 +
 automation/scripts/build                | 127 ++++++++++++++------------------
 5 files changed, 59 insertions(+), 79 deletions(-)
 delete mode 100644 automation/configs/x86/hvm_only_config
 delete mode 100644 automation/configs/x86/no_hvm_pv_config
 delete mode 100644 automation/configs/x86/pv_only_config
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/6 CI: Drop automation/configs/

Having 3 extra hypervisor builds on the end of a full build is deeply confusing to debug if one of them fails, because the .config file presented in the artefacts is not the one which caused a build failure. Also, the log tends to be truncated in the UI.

PV-only is tested as part of PV-Shim in a full build anyway, so doesn't need repeating. HVM-only and neither will come up frequently in randconfig, so drop all the logic here to simplify things.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/configs/x86/hvm_only_config  |  3 ---
 automation/configs/x86/no_hvm_pv_config |  3 ---
 automation/configs/x86/pv_only_config   |  3 ---
 automation/scripts/build                | 21 ---------------------
 4 files changed, 30 deletions(-)
 delete mode 100644 automation/configs/x86/hvm_only_config
 delete mode 100644 automation/configs/x86/no_hvm_pv_config
 delete mode 100644 automation/configs/x86/pv_only_config
automation/configs/x86/hvm_only_config deleted +0 -3
deleted file mode 100644
index 9efbddd5353b..000000000000
@@ -1,3 +0,0 @@
-CONFIG_HVM=y
-# CONFIG_PV is not set
-# CONFIG_DEBUG is not set
automation/configs/x86/no_hvm_pv_config deleted +0 -3
deleted file mode 100644
index 0bf6a8e46818..000000000000
@@ -1,3 +0,0 @@
-# CONFIG_HVM is not set
-# CONFIG_PV is not set
-# CONFIG_DEBUG is not set
automation/configs/x86/pv_only_config deleted +0 -3
deleted file mode 100644
index e9d8b4a7c753..000000000000
@@ -1,3 +0,0 @@
-CONFIG_PV=y
-# CONFIG_HVM is not set
-# CONFIG_DEBUG is not set
automation/scripts/build +0 -21
index a5934190634b..5dafa72ba540 100755
@@ -85,24 +85,3 @@ if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then
         cp -r dist binaries/
     fi
 fi
-
-if [[ "${hypervisor_only}" == "y" ]]; then
-    # If we are build testing a specific Kconfig exit now, there's no point in
-    # testing all the possible configs.
-    exit 0
-fi
-
-# Build all the configs we care about
-case ${XEN_TARGET_ARCH} in
-    x86_64) arch=x86 ;;
-    *) exit 0 ;;
-esac
-
-cfg_dir="automation/configs/${arch}"
-for cfg in `ls ${cfg_dir}`; do
-    echo "Building $cfg"
-    make -j$(nproc) -C xen clean
-    rm -f xen/.config
-    make -C xen KBUILD_DEFCONFIG=../../../../${cfg_dir}/${cfg} defconfig
-    make -j$(nproc) -C xen
-done
Signature · 1 line
2.11.0
Reviewed-by on v1 1/6
#2 lore raw

Re: [PATCH 1/6] CI: Drop automation/configs/

Hi Andrew,

Quoted 8 lines · Andrew Cooper

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Having 3 extra hypervisor builds on the end of a full build is deeply confusing to debug if one of them fails, because the .config file presented in the artefacts is not the one which caused a build failure. Also, the log tends to be truncated in the UI.

Quoted 5 lines

PV-only is tested as part of PV-Shim in a full build anyway, so doesn't need repeating. HVM-only and neither will come up frequently in randconfig, so drop all the logic here to simplify things.

~Michal

Reviewed-by on v1 1/6
#3 lore raw

Re: [PATCH 1/6] CI: Drop automation/configs/

In reply to Andrew Cooper
Quoted 10 lines · Andrew Cooper

Having 3 extra hypervisor builds on the end of a full build is deeply confusing to debug if one of them fails, because the .config file presented in the artefacts is not the one which caused a build failure. Also, the log tends to be truncated in the UI.

PV-only is tested as part of PV-Shim in a full build anyway, so doesn't need repeating. HVM-only and neither will come up frequently in randconfig, so drop all the logic here to simplify things.

Quoted 75 lines

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/configs/x86/hvm_only_config  |  3 ---
 automation/configs/x86/no_hvm_pv_config |  3 ---
 automation/configs/x86/pv_only_config   |  3 ---
 automation/scripts/build                | 21 ---------------------
 4 files changed, 30 deletions(-)
 delete mode 100644 automation/configs/x86/hvm_only_config
 delete mode 100644 automation/configs/x86/no_hvm_pv_config
 delete mode 100644 automation/configs/x86/pv_only_config
automation/configs/x86/hvm_only_config deleted +0 -3
deleted file mode 100644
index 9efbddd5353b..000000000000
@@ -1,3 +0,0 @@
-CONFIG_HVM=y
-# CONFIG_PV is not set
-# CONFIG_DEBUG is not set
automation/configs/x86/no_hvm_pv_config deleted +0 -3
deleted file mode 100644
index 0bf6a8e46818..000000000000
@@ -1,3 +0,0 @@
-# CONFIG_HVM is not set
-# CONFIG_PV is not set
-# CONFIG_DEBUG is not set
automation/configs/x86/pv_only_config deleted +0 -3
deleted file mode 100644
index e9d8b4a7c753..000000000000
@@ -1,3 +0,0 @@
-CONFIG_PV=y
-# CONFIG_HVM is not set
-# CONFIG_DEBUG is not set
automation/scripts/build +0 -21
index a5934190634b..5dafa72ba540 100755
@@ -85,24 +85,3 @@ if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then
         cp -r dist binaries/
     fi
 fi
-
-if [[ "${hypervisor_only}" == "y" ]]; then
-    # If we are build testing a specific Kconfig exit now, there's no point in
-    # testing all the possible configs.
-    exit 0
-fi
-
-# Build all the configs we care about
-case ${XEN_TARGET_ARCH} in
-    x86_64) arch=x86 ;;
-    *) exit 0 ;;
-esac
-
-cfg_dir="automation/configs/${arch}"
-for cfg in `ls ${cfg_dir}`; do
-    echo "Building $cfg"
-    make -j$(nproc) -C xen clean
-    rm -f xen/.config
-    make -C xen KBUILD_DEFCONFIG=../../../../${cfg_dir}/${cfg} defconfig
-    make -j$(nproc) -C xen
-done
Signature · 1 line
2.11.0
#4 lore raw

PATCH 2/6 CI: Remove guesswork about which artefacts to preserve

In reply to Andrew Cooper

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +14 -8
index 5dafa72ba540..8dee1cbbc251 100755
@@ -70,18 +70,24 @@ if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
     cfgargs+=("--with-system-seabios=/bin/false")
 fi
 
+# Directory for the artefacts to be dumped into
+mkdir binaries
+
 if [[ "${hypervisor_only}" == "y" ]]; then
+    # Xen-only build
     make -j$(nproc) xen
+
+    # Preserve artefacts
+    cp xen/xen binaries/xen
 else
+    # Full build
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
-fi
 
-# Extract artifacts to avoid getting rewritten by customised builds
-mkdir binaries
-if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then
-    cp xen/xen binaries/xen
-    if [[ "${hypervisor_only}" != "y" ]]; then
-        cp -r dist binaries/
-    fi
+    # Preserve artefacts
+    # Note: Some smoke tests depending on finding binaries/xen on a full build
+    # even though dist/ contains everything, while some containers don't even
+    # build Xen
+    cp -r dist binaries/
+    if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi
 fi
Signature · 1 line
2.11.0
Reviewed-by on v1 2/6
#5 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

Hi Andrew,

Quoted 6 lines · Andrew Cooper

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Quoted 7 lines

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

I'd prefer to keep using "artifacts" and not "artefacts" as the former is what GitLab uses and what we use in build/test.yaml.

Apart from that:

~Michal

#6 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

Quoted 12 lines · Michal Orzel

Hi Andrew,

Quoted 9 lines · Andrew Cooper

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

Quoted 6 lines
Quoted 4 lines

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

I'd prefer to keep using "artifacts" and not "artefacts" as the former is what GitLab uses and what we use in build/test.yaml.

Xen is written in British English.  We're forced to compromise for external dependencies, but xen.git is mostly British not American.

~Andrew

#7 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

Quoted 18 lines · Andrew Cooper
Quoted 15 lines · Michal Orzel

Hi Andrew,

Quoted 12 lines · Andrew Cooper

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I'd prefer to keep using "artifacts" and not "artefacts" as the former is what GitLab uses and what we use in build/test.yaml.

Xen is written in British English. We're forced to compromise for external dependencies, but xen.git is mostly British not American.

True, but from the consistency perspective and easy grepping, it is beneficial to stick to what a subsystem uses by default.

Quoted 2 lines

~Andrew

~Michal

Reviewed-by on v1 2/6
#8 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

In reply to Andrew Cooper
Quoted 46 lines · Andrew Cooper

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +8 -4
index 5dafa72ba540..8dee1cbbc251 100755
@@ -70,18 +70,24 @@ if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
     cfgargs+=("--with-system-seabios=/bin/false")
 fi
 
+# Directory for the artefacts to be dumped into
+mkdir binaries
+
 if [[ "${hypervisor_only}" == "y" ]]; then
+    # Xen-only build
     make -j$(nproc) xen
+
+    # Preserve artefacts
+    cp xen/xen binaries/xen
 else
+    # Full build
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
-fi
 
-# Extract artifacts to avoid getting rewritten by customised builds
-mkdir binaries
-if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then

automation/scripts/build if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then · line 92

Quoted 10 lines
automation/scripts/build +6 -4
-    cp xen/xen binaries/xen
-    if [[ "${hypervisor_only}" != "y" ]]; then
-        cp -r dist binaries/
-    fi
+    # Preserve artefacts
+    # Note: Some smoke tests depending on finding binaries/xen on a full build
+    # even though dist/ contains everything, while some containers don't even
+    # build Xen
+    cp -r dist binaries/
+    if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi

why the "if" ?

You could just:

cp xen/xen binaries/xen

unconditionally?

If you are OK with this change you can do it on commit

#9 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

Quoted 59 lines · Stefano Stabellini
Quoted 46 lines · Andrew Cooper

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +8 -4
index 5dafa72ba540..8dee1cbbc251 100755
@@ -70,18 +70,24 @@ if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
     cfgargs+=("--with-system-seabios=/bin/false")
 fi
 
+# Directory for the artefacts to be dumped into
+mkdir binaries
+
 if [[ "${hypervisor_only}" == "y" ]]; then
+    # Xen-only build
     make -j$(nproc) xen
+
+    # Preserve artefacts
+    cp xen/xen binaries/xen
 else
+    # Full build
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
-fi
 
-# Extract artifacts to avoid getting rewritten by customised builds
-mkdir binaries
-if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then

automation/scripts/build if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then · line 92

Quoted 10 lines
automation/scripts/build +6 -4
-    cp xen/xen binaries/xen
-    if [[ "${hypervisor_only}" != "y" ]]; then
-        cp -r dist binaries/
-    fi
+    # Preserve artefacts
+    # Note: Some smoke tests depending on finding binaries/xen on a full build
+    # even though dist/ contains everything, while some containers don't even
+    # build Xen
+    cp -r dist binaries/
+    if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi

why the "if" ?

Quoted 5 lines

You could just:

cp xen/xen binaries/xen

unconditionally?

No - the script is set -e and xen/xen doesn't exist in the Jessie32 containers.

That's why the old logic had an "if not x86_32" guard (except it also guarded the regular dist -> binaries/ copy which is problematic).

At a guess, the other working option would be cp xen/xen binaries/xen || :

I don't much care which of these two gets used, but pretty much anything else results in a failed test.

~Andrew

#10 lore raw

Re: [PATCH 2/6] CI: Remove guesswork about which artefacts to preserve

Quoted 69 lines · Andrew Cooper
Quoted 59 lines · Stefano Stabellini
Quoted 46 lines · Andrew Cooper

Preserve the artefacts based on the make rune we actually ran, rather than guesswork about which rune we would have run based on other settings.

Note that the ARM qemu smoke tests depend on finding binaries/xen even from full builds. Also that the Jessie-32 containers build tools but not Xen.

This means the x86_32 builds now store relevant artefacts. No change in other configurations.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +8 -4
index 5dafa72ba540..8dee1cbbc251 100755
@@ -70,18 +70,24 @@ if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
     cfgargs+=("--with-system-seabios=/bin/false")
 fi
 
+# Directory for the artefacts to be dumped into
+mkdir binaries
+
 if [[ "${hypervisor_only}" == "y" ]]; then
+    # Xen-only build
     make -j$(nproc) xen
+
+    # Preserve artefacts
+    cp xen/xen binaries/xen
 else
+    # Full build
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
-fi
 
-# Extract artifacts to avoid getting rewritten by customised builds
-mkdir binaries
-if [[ "${XEN_TARGET_ARCH}" != "x86_32" ]]; then

automation/scripts/build if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then · line 92

Quoted 10 lines
automation/scripts/build +6 -4
-    cp xen/xen binaries/xen
-    if [[ "${hypervisor_only}" != "y" ]]; then
-        cp -r dist binaries/
-    fi
+    # Preserve artefacts
+    # Note: Some smoke tests depending on finding binaries/xen on a full build
+    # even though dist/ contains everything, while some containers don't even
+    # build Xen
+    cp -r dist binaries/
+    if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi

why the "if" ?

Quoted 5 lines

You could just:

cp xen/xen binaries/xen

unconditionally?

No - the script is set -e and xen/xen doesn't exist in the Jessie32 containers.

Quoted 7 lines

That's why the old logic had an "if not x86_32" guard (except it also guarded the regular dist -> binaries/ copy which is problematic).

At a guess, the other working option would be cp xen/xen binaries/xen || :

I don't much care which of these two gets used, but pretty much anything else results in a failed test.

I didn't realize that. I think your version is this patch is better, keep it as is

#11 lore raw

PATCH 3/6 CI: Only calculate ./configure args if needed

In reply to Andrew Cooper

This is purely code motion of the cfgargs construction, into the case where it is used.

No practical change.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 63 ++++++++++++++++++++++++------------------------
 1 file changed, 31 insertions(+), 32 deletions(-)
automation/scripts/build +31 -32
index 8dee1cbbc251..f2301d08789d 100755
@@ -39,37 +39,6 @@ if [[ "${XEN_TARGET_ARCH}" = "arm32" ]]; then
     hypervisor_only="y"
 fi
 
-# build up our configure options
-cfgargs=()
-cfgargs+=("--enable-docs")
-
-if [[ "${CC}" == "clang"* ]]; then
-    # SeaBIOS cannot be built with clang
-    cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
-    # iPXE cannot be built with clang
-    cfgargs+=("--with-system-ipxe=/usr/lib/ipxe/ipxe.pxe")
-    # newlib cannot be built with clang so we cannot build stubdoms
-    cfgargs+=("--disable-stubdom")
-fi
-
-if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
-    # disable --disable-werror for QEMUU when building with MUSL
-    cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
-    # SeaBIOS doesn't build on MUSL systems
-    cfgargs+=("--with-system-seabios=/bin/false")
-fi
-
-# Qemu requires Python 3.5 or later, and ninja
-if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
-        || ! type ninja; then
-    cfgargs+=("--with-system-qemu=/bin/false")
-fi
-
-# SeaBIOS requires GCC 4.6 or later
-if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
-    cfgargs+=("--with-system-seabios=/bin/false")
-fi
-
 # Directory for the artefacts to be dumped into
 mkdir binaries
 
@@ -80,7 +49,37 @@ if [[ "${hypervisor_only}" == "y" ]]; then
     # Preserve artefacts
     cp xen/xen binaries/xen
 else
-    # Full build
+    # Full build.  Figure out our ./configure options
+    cfgargs=()
+    cfgargs+=("--enable-docs")
+
+    if [[ "${CC}" == "clang"* ]]; then
+        # SeaBIOS cannot be built with clang
+        cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
+        # iPXE cannot be built with clang
+        cfgargs+=("--with-system-ipxe=/usr/lib/ipxe/ipxe.pxe")
+        # newlib cannot be built with clang so we cannot build stubdoms
+        cfgargs+=("--disable-stubdom")
+    fi
+
+    if  ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
+        # disable --disable-werror for QEMUU when building with MUSL
+        cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
+        # SeaBIOS doesn't build on MUSL systems
+        cfgargs+=("--with-system-seabios=/bin/false")
+    fi
+
+    # Qemu requires Python 3.5 or later, and ninja
+    if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
+            || ! type ninja; then
+        cfgargs+=("--with-system-qemu=/bin/false")
+    fi
+
+    # SeaBIOS requires GCC 4.6 or later
+    if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
+        cfgargs+=("--with-system-seabios=/bin/false")
+    fi
+
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
 
Signature · 1 line
2.11.0
Reviewed-by on v1 3/6
#13 lore raw

Re: [PATCH 3/6] CI: Only calculate ./configure args if needed

In reply to Andrew Cooper
Quoted 6 lines · Andrew Cooper

This is purely code motion of the cfgargs construction, into the case where it is used.

No practical change.

Quoted 94 lines

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 63 ++++++++++++++++++++++++------------------------
 1 file changed, 31 insertions(+), 32 deletions(-)
automation/scripts/build +31 -32
index 8dee1cbbc251..f2301d08789d 100755
@@ -39,37 +39,6 @@ if [[ "${XEN_TARGET_ARCH}" = "arm32" ]]; then
     hypervisor_only="y"
 fi
 
-# build up our configure options
-cfgargs=()
-cfgargs+=("--enable-docs")
-
-if [[ "${CC}" == "clang"* ]]; then
-    # SeaBIOS cannot be built with clang
-    cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
-    # iPXE cannot be built with clang
-    cfgargs+=("--with-system-ipxe=/usr/lib/ipxe/ipxe.pxe")
-    # newlib cannot be built with clang so we cannot build stubdoms
-    cfgargs+=("--disable-stubdom")
-fi
-
-if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
-    # disable --disable-werror for QEMUU when building with MUSL
-    cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
-    # SeaBIOS doesn't build on MUSL systems
-    cfgargs+=("--with-system-seabios=/bin/false")
-fi
-
-# Qemu requires Python 3.5 or later, and ninja
-if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
-        || ! type ninja; then
-    cfgargs+=("--with-system-qemu=/bin/false")
-fi
-
-# SeaBIOS requires GCC 4.6 or later
-if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
-    cfgargs+=("--with-system-seabios=/bin/false")
-fi
-
 # Directory for the artefacts to be dumped into
 mkdir binaries
 
@@ -80,7 +49,37 @@ if [[ "${hypervisor_only}" == "y" ]]; then
     # Preserve artefacts
     cp xen/xen binaries/xen
 else
-    # Full build
+    # Full build.  Figure out our ./configure options
+    cfgargs=()
+    cfgargs+=("--enable-docs")
+
+    if [[ "${CC}" == "clang"* ]]; then
+        # SeaBIOS cannot be built with clang
+        cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
+        # iPXE cannot be built with clang
+        cfgargs+=("--with-system-ipxe=/usr/lib/ipxe/ipxe.pxe")
+        # newlib cannot be built with clang so we cannot build stubdoms
+        cfgargs+=("--disable-stubdom")
+    fi
+
+    if  ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
+        # disable --disable-werror for QEMUU when building with MUSL
+        cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
+        # SeaBIOS doesn't build on MUSL systems
+        cfgargs+=("--with-system-seabios=/bin/false")
+    fi
+
+    # Qemu requires Python 3.5 or later, and ninja
+    if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
+            || ! type ninja; then
+        cfgargs+=("--with-system-qemu=/bin/false")
+    fi
+
+    # SeaBIOS requires GCC 4.6 or later
+    if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
+        cfgargs+=("--with-system-seabios=/bin/false")
+    fi
+
     ./configure "${cfgargs[@]}"
     make -j$(nproc) dist
 
Signature · 1 line
2.11.0
#14 lore raw

PATCH 4/6 CI: Express HYPERVISOR_ONLY in build.yml

In reply to Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +2 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
 debian-unstable-gcc-arm32-randconfig:
   extends: .gcc-arm32-cross-build
automation/scripts/build +4 -7
index f2301d08789d..4c6d1f3b70bc 100755
@@ -19,7 +19,9 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
     fi
 
     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
-    hypervisor_only="y"
+
+    # RANDCONFIG implies HYPERVISOR_ONLY
+    HYPERVISOR_ONLY="y"
 else
     echo "CONFIG_DEBUG=${debug}" > xen/.config
 
@@ -34,15 +36,10 @@ fi
 # to exit early -- bash is invoked with -e.
 cp xen/.config xen-config
 
-# arm32 only cross-compiles the hypervisor
-if [[ "${XEN_TARGET_ARCH}" = "arm32" ]]; then
-    hypervisor_only="y"
-fi
-
 # Directory for the artefacts to be dumped into
 mkdir binaries
 
-if [[ "${hypervisor_only}" == "y" ]]; then
+if [[ "${HYPERVISOR_ONLY}" == "y" ]]; then
     # Xen-only build
     make -j$(nproc) xen
 
Signature · 1 line
2.11.0
#15 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Hi Andrew,

This is not something critical but I would like to see HYPERVISOR_ONLY set explicitly for all debian-unstable-gcc-arm32-* jobs as it is not clear for end-user of build.yaml that "the RANDCONFIG -> HYPERVISOR_ONLY implication."

~Oleksii

Quoted 75 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase.  It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable- gcc-arm32-*
testcases at the point that arm32 get matched with a container that can only
build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>
---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------

 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab- renamed from automation/gitlab-ci/build.yaml +0 -0

ci/build.yaml
index 93d9ff69a9f2..e6a9357de3ef 100644

automation/gitlab-ci/build.yaml +0 -0
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:

   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

 debian-unstable-gcc-arm32-randconfig:
   extends: .gcc-arm32-cross-build

automation/scripts/build +0 -0
index f2301d08789d..4c6d1f3b70bc 100755
@@ -19,7 +19,9 @@ if [[ "${RANDCONFIG}" == "y" ]]; then

     fi

     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig -    hypervisor_only="y"
+
+    # RANDCONFIG implies HYPERVISOR_ONLY
+    HYPERVISOR_ONLY="y"
 else
     echo "CONFIG_DEBUG=${debug}" > xen/.config

automation/scripts/build +0 -0
@@ -34,15 +36,10 @@ fi
  # to exit early -- bash is invoked with -e.

 cp xen/.config xen-config

automation/scripts/build +0 -5
-# arm32 only cross-compiles the hypervisor
-if [[ "${XEN_TARGET_ARCH}" = "arm32" ]]; then
-    hypervisor_only="y"
-fi
-

 # Directory for the artefacts to be dumped into
 mkdir binaries

automation/scripts/build +1 -1
-if [[ "${hypervisor_only}" == "y" ]]; then
+if [[ "${HYPERVISOR_ONLY}" == "y" ]]; then

     # Xen-only build
     make -j$(nproc) xen

Reviewed-by on v1 4/6
#16 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

In reply to Andrew Cooper

Hi Andrew,

Quoted 5 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Quoted 7 lines

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

With regards to Oleksii comment, I do not think we should add HYPERVISOR_ONLY explicitly for randconfig debian-unstable-gcc-arm32-* jobs. Doing so would require similar change for all the randconfig jobs to be consistent and this is not needed.

~Michal

#17 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

In reply to Andrew Cooper
Quoted 26 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +0 -0
index 93d9ff69a9f2..e6a9357de3ef 100644

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 10 lines
automation/gitlab-ci/build.yaml +2 -0
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 38 lines
automation/gitlab-ci/build.yaml +0 -0
 debian-unstable-gcc-arm32-randconfig:
   extends: .gcc-arm32-cross-build
automation/scripts/build +4 -7
index f2301d08789d..4c6d1f3b70bc 100755
@@ -19,7 +19,9 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
     fi
 
     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
-    hypervisor_only="y"
+
+    # RANDCONFIG implies HYPERVISOR_ONLY
+    HYPERVISOR_ONLY="y"
 else
     echo "CONFIG_DEBUG=${debug}" > xen/.config
 
@@ -34,15 +36,10 @@ fi
 # to exit early -- bash is invoked with -e.
 cp xen/.config xen-config
 
-# arm32 only cross-compiles the hypervisor
-if [[ "${XEN_TARGET_ARCH}" = "arm32" ]]; then
-    hypervisor_only="y"
-fi
-
 # Directory for the artefacts to be dumped into
 mkdir binaries
 
-if [[ "${hypervisor_only}" == "y" ]]; then
+if [[ "${HYPERVISOR_ONLY}" == "y" ]]; then
     # Xen-only build
     make -j$(nproc) xen
 
Signature · 1 line
2.11.0
#18 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

~Andrew

#19 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 46 lines · Andrew Cooper
Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 8 lines

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

If it is a property of the container, shouldn't HYPERVISOR_ONLY be set every time the debian:unstable-arm32-gcc container is used? Including debian-unstable-gcc-arm32-randconfig and debian-unstable-gcc-arm32-debug-randconfig?

I realize that the other 2 jobs are randconfigs so HYPERVISOR_ONLY gets set anyway. But if HYPERVISOR_ONLY is a property of the specific container, then I think it would be best to be consistent and set HYPERVISOR_ONLY everywhere debian:unstable-arm32-gcc is used.

E.g. one day we could just randconfigs to build also the tools with a simple change to the build script and otherwise we would need to remember to also add the HYPERVISOR_ONLY tag for the other 2 jobs using debian:unstable-arm32-gcc.

#20 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 63 lines · Stefano Stabellini
Quoted 44 lines · Andrew Cooper
Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 8 lines

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

If it is a property of the container, shouldn't HYPERVISOR_ONLY be set every time the debian:unstable-arm32-gcc container is used? Including debian-unstable-gcc-arm32-randconfig and debian-unstable-gcc-arm32-debug-randconfig?

I realize that the other 2 jobs are randconfigs so HYPERVISOR_ONLY gets set anyway. But if HYPERVISOR_ONLY is a property of the specific container, then I think it would be best to be consistent and set HYPERVISOR_ONLY everywhere debian:unstable-arm32-gcc is used.

Quoted 4 lines

E.g. one day we could just randconfigs to build also the tools with a simple change to the build script and otherwise we would need to remember to also add the HYPERVISOR_ONLY tag for the other 2 jobs using debian:unstable-arm32-gcc.

Ok, so we want 4 HYPERVISOR_ONLY's in total, one for each instance of
CONTAINER: debian:unstable-arm32-gcc ?

~Andrew

#21 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 64 lines · Andrew Cooper
Quoted 63 lines · Stefano Stabellini
Quoted 44 lines · Andrew Cooper
Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 8 lines

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

If it is a property of the container, shouldn't HYPERVISOR_ONLY be set every time the debian:unstable-arm32-gcc container is used? Including debian-unstable-gcc-arm32-randconfig and debian-unstable-gcc-arm32-debug-randconfig?

I realize that the other 2 jobs are randconfigs so HYPERVISOR_ONLY gets set anyway. But if HYPERVISOR_ONLY is a property of the specific container, then I think it would be best to be consistent and set HYPERVISOR_ONLY everywhere debian:unstable-arm32-gcc is used.

Quoted 7 lines
Quoted 4 lines

E.g. one day we could just randconfigs to build also the tools with a simple change to the build script and otherwise we would need to remember to also add the HYPERVISOR_ONLY tag for the other 2 jobs using debian:unstable-arm32-gcc.

Ok, so we want 4 HYPERVISOR_ONLY's in total, one for each instance of
CONTAINER: debian:unstable-arm32-gcc ?

yeah

#22 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 65 lines · Stefano Stabellini
Quoted 64 lines · Andrew Cooper
Quoted 63 lines · Stefano Stabellini
Quoted 44 lines · Andrew Cooper
Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 8 lines

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

If it is a property of the container, shouldn't HYPERVISOR_ONLY be set every time the debian:unstable-arm32-gcc container is used? Including debian-unstable-gcc-arm32-randconfig and debian-unstable-gcc-arm32-debug-randconfig?

I realize that the other 2 jobs are randconfigs so HYPERVISOR_ONLY gets set anyway. But if HYPERVISOR_ONLY is a property of the specific container, then I think it would be best to be consistent and set HYPERVISOR_ONLY everywhere debian:unstable-arm32-gcc is used.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

E.g. one day we could just randconfigs to build also the tools with a simple change to the build script and otherwise we would need to remember to also add the HYPERVISOR_ONLY tag for the other 2 jobs using debian:unstable-arm32-gcc.

Ok, so we want 4 HYPERVISOR_ONLY's in total, one for each instance of
CONTAINER: debian:unstable-arm32-gcc ?

yeah

Can I take that as an R-by/A-by then?

~Andrew

#23 lore raw

Re: [PATCH 4/6] CI: Express HYPERVISOR_ONLY in build.yml

Quoted 70 lines · Andrew Cooper
Quoted 69 lines · Stefano Stabellini
Quoted 68 lines · Andrew Cooper
Quoted 67 lines · Stefano Stabellini
Quoted 44 lines · Andrew Cooper
Quoted 33 lines · Stefano Stabellini
Quoted 31 lines · Andrew Cooper

Whether to build only Xen, or everything, is a property of container, toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.

Capitalise HYPERVISOR_ONLY and have it set by the debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with a container that can only build Xen.

For simplicity, retain the RANDCONFIG -> HYPERVISOR_ONLY implication.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/gitlab-ci/build.yaml |  2 ++
 automation/scripts/build        | 11 ++++-------
 2 files changed, 6 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +1 -0
index 93d9ff69a9f2..e6a9357de3ef 100644
@@ -516,11 +516,13 @@ debian-unstable-gcc-arm32:
   extends: .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y
 
Quoted 6 lines

automation/gitlab-ci/build.yaml debian-unstable-gcc-arm32: · line 525

Quoted 5 lines
automation/gitlab-ci/build.yaml +1 -0
 debian-unstable-gcc-arm32-debug:
   extends: .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm32-gcc
+    HYPERVISOR_ONLY: y

can you move the setting of HYPERVISOR_ONLY to .arm32-cross-build-tmpl ?

Not really - that's the point I'm trying to make in the commit message.

Quoted 2 lines

I think that makes the most sense because .arm32-cross-build-tmpl is the one setting XEN_TARGET_ARCH and also the x86_64 tag.

Quoted 8 lines

It's not about x86_64; its about the container.

Whether we can build just Xen, or everything, solely depends on the contents in debian:unstable-arm32-gcc

If we wanted to, we could update unstable-arm32-gcc's dockerfile to install the arm32 cross user libs, and drop this HYPERVISOR_ONLY restriction.

If it is a property of the container, shouldn't HYPERVISOR_ONLY be set every time the debian:unstable-arm32-gcc container is used? Including debian-unstable-gcc-arm32-randconfig and debian-unstable-gcc-arm32-debug-randconfig?

I realize that the other 2 jobs are randconfigs so HYPERVISOR_ONLY gets set anyway. But if HYPERVISOR_ONLY is a property of the specific container, then I think it would be best to be consistent and set HYPERVISOR_ONLY everywhere debian:unstable-arm32-gcc is used.

E.g. one day we could just randconfigs to build also the tools with a simple change to the build script and otherwise we would need to remember to also add the HYPERVISOR_ONLY tag for the other 2 jobs using debian:unstable-arm32-gcc.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

Ok, so we want 4 HYPERVISOR_ONLY's in total, one for each instance of
CONTAINER: debian:unstable-arm32-gcc ?

yeah

Can I take that as an R-by/A-by then?

yep

#24 lore raw

PATCH 5/6 CI: Fix build script when CROSS_COMPILE is in use

In reply to Andrew Cooper

Some testcases use a cross compiler. Presently it's only arm32 and due to previous cleanup the only thing which is now wrong is printing the compiler version at the start of day.

Construct $cc to match what make will eventually choose given CROSS_COMPILE, taking care not to modify $CC. Use $cc throughout the rest of the script.

Also correct the compiler detection logic. Plain "gcc" was wrong, and "clang"* was a bodge highlighting the issue, but neither survive the CROSS_COMPILE correction. Instead, construct cc_is_{gcc,clang} booleans like we do elsewhere in the build system, by querying the --version text for gcc or clang.

While making this change, adjust cc_ver to be calculated once at the same time as cc_is_* are calculated.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +14 -8
index 4c6d1f3b70bc..206312ecc7d0 100755
@@ -2,13 +2,12 @@
 
 test -f /etc/os-release && cat "$_"
 
-$CC --version
+# Construct $cc such that it matches what `make` will chose when taking
+# CROSS_COMPILE into account.  Do not modify $CC directly, as that will cause
+# `make` to double-account CROSS_COMPILE.
+cc="${CROSS_COMPILE}${CC}"
 
-# Express the compiler version as an integer.  e.g. GCC 4.9.2 => 0x040902
-cc-ver()
-{
-    $CC -dumpversion | awk -F. '{ printf "0x%02x%02x%02x", $1, $2, $3 }'
-}
+$cc --version
 
 # random config or default config
 if [[ "${RANDCONFIG}" == "y" ]]; then
@@ -50,7 +49,14 @@ else
     cfgargs=()
     cfgargs+=("--enable-docs")
 
-    if [[ "${CC}" == "clang"* ]]; then
+    # booleans for which compiler is in use
+    cc_is_gcc="$($cc --version | grep -q gcc && echo "y" || :)"
+    cc_is_clang="$($cc --version | grep -q clang && echo "y" || :)"
+
+    # The compiler version as an integer.  e.g. GCC 4.9.2 => 0x040902
+    cc_ver="$($cc -dumpversion | awk -F. '{ printf "0x%02x%02x%02x", $1, $2, $3 }')"
+
+    if [[ "${cc_is_clang}" == "y" ]]; then
         # SeaBIOS cannot be built with clang
         cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
         # iPXE cannot be built with clang
@@ -73,7 +79,7 @@ else
     fi
 
     # SeaBIOS requires GCC 4.6 or later
-    if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
+    if [[ "${cc_is_gcc}" == "y" && "${cc_ver}" -lt 0x040600 ]]; then
         cfgargs+=("--with-system-seabios=/bin/false")
     fi
 
Signature · 1 line
2.11.0
Reviewed-by on v1 5/6
#25 lore raw

Re: [PATCH 5/6] CI: Fix build script when CROSS_COMPILE is in use

Quoted 13 lines · Andrew Cooper

Some testcases use a cross compiler. Presently it's only arm32 and due to previous cleanup the only thing which is now wrong is printing the compiler version at the start of day.

Construct $cc to match what make will eventually choose given CROSS_COMPILE, taking care not to modify $CC. Use $cc throughout the rest of the script.

Also correct the compiler detection logic. Plain "gcc" was wrong, and "clang"* was a bodge highlighting the issue, but neither survive the CROSS_COMPILE correction. Instead, construct cc_is_{gcc,clang} booleans like we do elsewhere in the build system, by querying the --version text for gcc or clang.

Quoted 4 lines

While making this change, adjust cc_ver to be calculated once at the same time as cc_is_* are calculated.

Quoted 61 lines

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
automation/scripts/build +14 -8
index 4c6d1f3b70bc..206312ecc7d0 100755
@@ -2,13 +2,12 @@
 
 test -f /etc/os-release && cat "$_"
 
-$CC --version
+# Construct $cc such that it matches what `make` will chose when taking
+# CROSS_COMPILE into account.  Do not modify $CC directly, as that will cause
+# `make` to double-account CROSS_COMPILE.
+cc="${CROSS_COMPILE}${CC}"
 
-# Express the compiler version as an integer.  e.g. GCC 4.9.2 => 0x040902
-cc-ver()
-{
-    $CC -dumpversion | awk -F. '{ printf "0x%02x%02x%02x", $1, $2, $3 }'
-}
+$cc --version
 
 # random config or default config
 if [[ "${RANDCONFIG}" == "y" ]]; then
@@ -50,7 +49,14 @@ else
     cfgargs=()
     cfgargs+=("--enable-docs")
 
-    if [[ "${CC}" == "clang"* ]]; then
+    # booleans for which compiler is in use
+    cc_is_gcc="$($cc --version | grep -q gcc && echo "y" || :)"
+    cc_is_clang="$($cc --version | grep -q clang && echo "y" || :)"
+
+    # The compiler version as an integer.  e.g. GCC 4.9.2 => 0x040902
+    cc_ver="$($cc -dumpversion | awk -F. '{ printf "0x%02x%02x%02x", $1, $2, $3 }')"
+
+    if [[ "${cc_is_clang}" == "y" ]]; then
         # SeaBIOS cannot be built with clang
         cfgargs+=("--with-system-seabios=/usr/share/seabios/bios.bin")
         # iPXE cannot be built with clang
@@ -73,7 +79,7 @@ else
     fi
 
     # SeaBIOS requires GCC 4.6 or later
-    if [[ "${CC}" == "gcc" && `cc-ver` -lt 0x040600 ]]; then
+    if [[ "${cc_is_gcc}" == "y" && "${cc_ver}" -lt 0x040600 ]]; then
         cfgargs+=("--with-system-seabios=/bin/false")
     fi
 
Signature · 1 line
2.11.0
#26 lore raw

PATCH 6/6 CI: Simplify the MUSL check

In reply to Andrew Cooper

There's no need to do ad-hoc string parsing. Use grep -q instead.

No functional change.

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
automation/scripts/build +1 -1
index 206312ecc7d0..f2f5e55bc04f 100755
@@ -65,7 +65,7 @@ else
         cfgargs+=("--disable-stubdom")
     fi
 
-    if  ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
+    if ldd /bin/ls | grep -q musl; then
         # disable --disable-werror for QEMUU when building with MUSL
         cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
         # SeaBIOS doesn't build on MUSL systems
Signature · 1 line
2.11.0
Reviewed-by on v1 6/6
#27 lore raw

Re: [PATCH 6/6] CI: Simplify the MUSL check

Quoted 5 lines · Andrew Cooper

There's no need to do ad-hoc string parsing. Use grep -q instead.

No functional change.

Quoted 26 lines

---
CC: Doug Goldstein <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Anthony PERARD <[email protected]>
CC: Michal Orzel <[email protected]>
CC: Oleksii Kurochko <[email protected]>

---
 automation/scripts/build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
automation/scripts/build +1 -1
index 206312ecc7d0..f2f5e55bc04f 100755
@@ -65,7 +65,7 @@ else
         cfgargs+=("--disable-stubdom")
     fi
 
-    if  ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
+    if ldd /bin/ls | grep -q musl; then
         # disable --disable-werror for QEMUU when building with MUSL
         cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
         # SeaBIOS doesn't build on MUSL systems
Signature · 1 line
2.11.0