Skip to content

Add support for local automated testing

Glenn Washburn grub-devel 2 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest3 patches 0 repliesNew
  2. v13 patches 0 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3scripts: Add general scripts to aid automated testingscripts/ci/build.sh scripts/ci/functions.sh scripts/ci/make-images.sh +2 more+418 -00 replies
2/3scripts: Add functions for CI stages and default variables to functions.shscripts/ci/functions.sh+922 -10 replies
3/3scripts: Add local-tester.sh script to run local CI testsscripts/ci/functions.local.sh scripts/local-tester.sh+76 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [LOCAL-CI v2 0/3] Add support for local automated testing

Updates since v1:
* update multipipe to return error code from children
* fix bootstrap run cache file not written to correct location
* fix run_test and run_build to properly return test and build error codes
* fix functional test post-processing to recognize failures in any functional test, not just ones whose name ends in "test"
* fix issue where tests which were scripts that didn't use @BUILD_SHEBANG@ as shell were being run with @BUILD_SHEBANG@ shell, and thus weren't running those tests properly (or at all)
* other minor improvements

This patch series aims to make automated testing of all supported targets easy. Not all targets are actually tested for a variety of reasons, but this series covers a significant portion of the target space and should make it easy to add more targets as we figure out how do run their tests. All supported targets (that I know of) are build tested.

The main work horse is the scripts/local-tester.sh script which ties everything together. It is very configurable, but defaults to building and testing as much as it can. To start it just run "./scripts/local-tester.sh" from the root of the repository. By default a directory named 'grub-tests' in the current working directory is created and everything is put in there. See the beginning of ./scripts/ci/function.sh for available environment variables that can be used to configure how you want it to run.

The file ./scripts/ci/function.sh has a bunch of functions most of which are for stages in the automated testing. It is intended to be sourced by config files for other CI systems so that they can reuse this code.

One might ask, why not just have all CI systems use local-tester.sh and put everything in there? The issue is that currently local-tester.sh does not do things in parallel (the make subprocess can be made more parallel with the JOBS environment variable). So in many CI systems, one could have all targets building and testing at the same time, which local-tester.sh only does this serially. This makes local-tester.sh a lot slower than it needs to be. Also other CI systems allow the CI pipeline to be broken into many stages, each of which could be run on a different machine, with the ability to cache the output of certain stages.

These scripts have been written and tested on debian based systems, specifically the reference system, Debian 11. Some of the stages are debian or debian- derivative specific, such as the package install stage which uses apt and dpkg. Most of the stages, I believe, are fairly distro agnostic and the ones that aren't should be able to be adapted for a specific distro fairly easily.

Also, this patch series is meant to be used on top of the grub-shell patch series already submitted to the list. It will run without that series, but some of the features may not work or work as well. Noteably, the QEMU tests for targets i386-efi, arm-efi and arm64-efi will fail.

Of particular note, there are some knobs that can provide a lot debugging output and save the intermediate files of failed tests for later analysis.

local-tester.sh will try to download and install all packages it needs to function. Obviously, this will not work when not running as a privileged user. A further patch series is intended, which will add support for running the system successfully completely as an unprivileged user. If local-tester.sh is run as an unprivileged user, it will skip running of privileged commands, like the package installer. This means it can continue past the package install phase, but it assumes that the needed packages are already installed.

Glenn

Glenn Washburn (3):
scripts: Add general scripts to aid automated testing
scripts: Add functions for CI stages and default variables to
functions.sh
scripts: Add local-tester.sh script to run local CI tests

 scripts/ci/build.sh           |  67 +++
 scripts/ci/functions.local.sh |  37 ++
 scripts/ci/functions.sh       | 954 ++++++++++++++++++++++++++++++++++
 scripts/ci/make-images.sh     |  86 +++
 scripts/ci/process-tests.sh   | 111 ++++
 scripts/ci/test.sh            | 121 +++++
 scripts/local-tester.sh       |  39 ++
 7 files changed, 1415 insertions(+)
 create mode 100755 scripts/ci/build.sh
 create mode 100644 scripts/ci/functions.local.sh
 create mode 100644 scripts/ci/functions.sh
 create mode 100755 scripts/ci/make-images.sh
 create mode 100755 scripts/ci/process-tests.sh
 create mode 100755 scripts/ci/test.sh
 create mode 100755 scripts/local-tester.sh

Interdiff against v1:

scripts/ci/functions.sh +45 -21
index f94f90dc1..734b36d3f 100644
@@ -350,7 +350,8 @@ EXPECTED_FAILURES=${EXPECTED_FAILURES:-"
       ### Usually this test passes, but occasionally the virtual machine is
       ### too slow and the time delta threshold is exceeded. So allow failures.
       #grub_cmd_sleep
-      ### This fails on the volume label check
+      ### This can fail on the volume label check because newer Debian versions
+      ### hfsprogs has a bug where mkfs.hfs does not write the volume label.
       hfs_test
       ### This fails because grub-fstest ls -la is not outputting file time
       ### on files, but it is on directories. In linux there are file times.
@@ -377,7 +378,6 @@ EXPECTED_FAILURES=${EXPECTED_FAILURES:-"
 
 EXPECTED_FUNCTIONAL_FAILURES=${EXPECTED_FUNCTIONAL_FAILURES:-"
       ### These have not worked for a very long time
-      videotest_checksum
       gfxterm_menu
       cmdline_cat_test
       ### Sometimes the machine the test are running on are slow causing
@@ -476,9 +476,17 @@ helper_process_redirects() {
 # that stderr is piped into the next command in the pipe as fd 3.
 multipipe() {
   local ADELIM=$'\x01'
-  local LAST_PID IFS_OLD CMD
+  local ERROR_ANY RET_PID RET_PID_NUM IFS_OLD CMD
   local PIPEFILEOUT PIPEFILEERR PIPEFILEOUTPREV PIPEFILEERRPREV
 
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -e) ERROR_ANY=1; shift;;
+      -r) RET_PID_NUM="$2"; shift 2;;
+      *) break;;
+    esac
+  done
+
   # For each command from last to first, start command with input and output
   # as file descriptors to named pipes.
   local i=0 max=$(echo "$*" | sed 's/|/\n/g' | wc -l) PID=
@@ -492,9 +500,10 @@ multipipe() {
       PIPEFILEERR=$(mktemp -u "${TMPDIR:-/tmp}"/multipipe.XXXXXXXXXX.fifo)
       mkfifo $PIPEFILEERR
       helper_process_redirects "$@" <$PIPEFILEOUT 3<$PIPEFILEERR &
-      LAST_PID=$!
+      PID=$!
     elif [ "$i" -eq "$((max-1))" ]; then
       helper_process_redirects "$@" >$PIPEFILEOUT 2>$PIPEFILEERR &
+      PID=$!
       rm -f $PIPEFILEOUT $PIPEFILEERR
     else
       PIPEFILEOUTPREV=$PIPEFILEOUT
@@ -517,13 +526,27 @@ multipipe() {
         # previous command in the pipeline.
         exec >$PIPEFILEOUTPREV 2>$PIPEFILEERRPREV
         helper_process_redirects "$@" <$PIPEFILEOUT 3<$PIPEFILEERR &
+        exit $!
       )
+      PID=$?
       rm -f $PIPEFILEOUTPREV $PIPEFILEERRPREV
     fi
+
+    if [ "$ERROR_ANY" = "1" ]; then
+      RET_PID="$RET_PID $PID"
+    elif [ -z "$RET_PID_NUM" ] || [ "$RET_PID_NUM" -eq "$((max-$i))" ]; then
+      RET_PID=$PID
+    elif [ "$RET_PID_NUM" -eq "$((max-$i))" ]; then
+      RET_PID=$PID
+    fi
+
     i=$((i+1))
     IFS=$IFSOLD
   done < <(IFS=$ADELIM; { echo -n "$*" | sed "s/${ADELIM}|${ADELIM}/\n/g"; echo; } | tac)
-  wait $LAST_PID
+
+  for PID in $RET_PID; do
+    wait -f $PID || return $?
+  done
 }
 
 multipipe_test() {
@@ -561,12 +584,8 @@ setup_packages() {
     echo "No apt binary found, skipping package installation..."
     return
   fi
-  if [ "$EUID" = 0 ]; then
-    :
-  elif which apt-user >/dev/null 2>&1; then
-    apt() { ROOT="rootfs.ovl" apt-user "$@" ||:; }
-  else
-    echo "Not root and apt-user not found, continuing assuming that all needed packages are installed..."
+  if [ "$EUID" != 0 ]; then
+    echo "Not root, continuing assuming that all needed packages are installed..."
     return
   fi
   apt $APT_OPTS update -yqq &&
@@ -584,6 +603,7 @@ setup_repo() {
     while [ ! -d "${GIT_REPO_DEFAULT}/.git" ]; do
       GIT_REPO_DEFAULT="${GIT_REPO_DEFAULT%/*}"
     done
+    GIT_REPO_DEFAULT="file://${GIT_REPO_DEFAULT}"
   fi
 
   if [ -d "$SRCDIR" ]; then
@@ -714,7 +734,7 @@ setup_bootstrap() {
   # We cache the bootstrap, so if the bootstrap canary exists, don't run the bootstrap
   if [ ! -f "$SRCDIR"/.bootstrap.finished ]; then
     ( cd "$SRCDIR" && ./bootstrap ) &&
-      touch .bootstrap.finished || RET=1
+      touch "$SRCDIR"/.bootstrap.finished || RET=1
   fi
   end_log -n "bootstrap"
   return ${RET:-0}
@@ -780,6 +800,7 @@ setup_env() {
 }
 
 run_build() {
+  local RET=0
   if [ "x${DISABLE_ALL_BUILDS}" = "xy" ]; then
     echo "All builds are disabled"
     return
@@ -803,7 +824,7 @@ run_build() {
 #!$SHELL
 if [ "x\${SHELL_TRACE}" = "xy" ]; then
   set -x
-  if [ -n "\${SHELL_OPTS##* -x*}" ]; then
+  if [ -z "\$SHELL_OPTS" ] || [ -n "\${SHELL_OPTS##* -x*}" ]; then
     SHELL_OPTS="\$SHELL_OPTS -x"
   fi
 fi
@@ -815,15 +836,17 @@ EOF
   export CONFIGURE_OPTS="${CONFIGURE_OPTS} CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh"
 
   if type multipipe >/dev/null; then
-    multipipe "${CI_SCRIPTS_DIR}/build.sh" \
-      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/build.log"
+    multipipe -e "${CI_SCRIPTS_DIR}/build.sh" \
+      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/build.log" || RET=$?
   else
-    { "${CI_SCRIPTS_DIR}/build.sh" |
-      tee >(exec cat >&3) >"${BUILDDIR}/build.log" 2>&1; } 3>&1
+    { { "${CI_SCRIPTS_DIR}/build.sh" || RET=$?; } |
+      tee >(exec cat >&3) >"${BUILDDIR}/build.log" 2>&1; } 3>&1 || :
   fi
+  return $RET
 }
 
 run_test() {
+  local RET=0
   if [ "x${DISABLE_ALL_TESTS}" = "xy" ]; then
     echo "All tests are disabled"
     return
@@ -831,12 +854,13 @@ run_test() {
 
   export TESTTMPDIR="${TMPDIR:-/tmp}/grub-test-$TARGET-${CI_COMMIT_SHORT_SHA}"
   if type multipipe >/dev/null; then
-    multipipe "${CI_SCRIPTS_DIR}/test.sh" \
-      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/test.log" || :
+    multipipe -e "${CI_SCRIPTS_DIR}/test.sh" \
+      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/test.log" || RET=$?
   else
-    { "${CI_SCRIPTS_DIR}/test.sh" |
-        tee >(exec cat >&3) >"${BUILDDIR}/test.log" 2>&1; } 3>&1 || :
+    { { "${CI_SCRIPTS_DIR}/test.sh" || RET=$?; } |
+      tee >(exec cat >&3) >"${BUILDDIR}/test.log" 2>&1; } 3>&1 || :
   fi
+  return $RET
 }
 
 post_processing() {
scripts/ci/make-images.sh +1 -1
index 3c5fd64f8..d156af46f 100755
@@ -74,7 +74,7 @@ for tfmt in $tformats; do
   [ "${_RET:-0}" -ne 0 ] && RET=$_RET
 
   if [ "$RET" -ne 0 ]; then
-    echo -e "${TXT_RED}"'Failed to build image for target format ${tfmt}:' "${TESTNAME}$TXT_CLEAR"
+    echo -e "${TXT_RED}Failed to build image for target format ${tfmt}: ${TESTNAME}$TXT_CLEAR"
     echo -e -n "$TXT_LOG_COLOR"
     echo "Last ${NUM_FAILED_LOG_LINES} lines of grub-mkimage verbose log"
     tail -n "${NUM_FAILED_LOG_LINES}" "${BUILDDIR}/grub-mkimage-${tfmt}.log"
scripts/ci/process-tests.sh +1 -1
index bdd6c652d..9bf763502 100755
@@ -66,7 +66,7 @@ if [ -f ${BUILDDIR}/test-suite.log ]; then
     # If any unexpected failures in the functional tests, count a failure in
     # grub_func_test as a true failure.
     elif [ "$TESTNAME" = "grub_func_test" ]; then
-      grep -E 'test: FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
+      grep -E ': FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
       while read FTESTNAME STATUS; do
         FTESTNAME=${FTESTNAME%:*}
         if echo "${EXPECTED_FUNCTIONAL_FAILURES}" | grep -qE "^\s*(${FTESTNAME}|${TARGET}:${FTESTNAME})$"; then
scripts/ci/test.sh +7 -6
index f70905462..833912587 100755
@@ -66,18 +66,19 @@ mkdir -p "\$TMPDIR"
 
 # if not a shell script, run normally
 if [ "\$(head -c2 \$1)" = "#!" ]; then
-  TEST_SHELL="$(head -n1 "${BUILDDIR}/grub-shell" | tail -c+3 | tr -d ' ')"
+  BUILD_SHEBANG="\$(head -n1 "${BUILDDIR}/grub-shell" | tail -c+3 | sed -E 's|^\\s*||')"
+  TEST_SHELL=\$(head -n1 "\$1" | tail -c+3 | sed -E 's|^\\s*||')
   # Only turn on tracing if the shell is the one used by grub-shell
-  if head -n1 \$1 | grep -q \$TEST_SHELL; then
+  if test "\${TEST_SHELL}" = "\${BUILD_SHEBANG}"; then
     if [ '(' "\${TEST_VERBOSITY:-0}" -gt 0 -o "x\${SHELL_TRACE}" = "xy" ')' \\
-        -a -n "\${TEST_SHELL##*-x*}" ]; then
-      TEST_SHELL="\$TEST_SHELL -x"
+        -a '(' -z "\$SHELL_OPTS" -o -n "\${SHELL_OPTS##*-x*}" ')' ]; then
+      SHELL_OPTS="\$SHELL_OPTS -x"
     fi
   fi
-fi;
+fi
 
 TSTART=\$(date +%s)
-eval \$(echo -n "\$STRACE" | sed "s/xxx/\$TESTNAME/g") \$TEST_SHELL "\$@" || RET=\$?
+eval \$(echo -n "\$STRACE" | sed "s/xxx/\$TESTNAME/g") "\$TEST_SHELL" "\$@" || RET=\$?
 TEND=\$(date +%s)
 echo "Test duration in seconds: \$((TEND - TSTART))"
 
Signature · 1 line
2.27.0