Skip to contentSkip to conversation

Allow easy re-running of grub-shell from failed tests

PATCH In review Glenn Washburn 2 participants 5 messages Jan 21, 2023 – Jan 25, 2023

lore
Share & tools
lore
#0 lore raw

cover Allow easy re-running of grub-shell from failed tests

When a non-native (eg. grub-shell/qemu) test fails it can be useful to be able to re-run the grub-shell invocation in at least two circumstances.

1. The tester wants to re-run the grub-shell invocation with extra args so that QEMU starts in GDB attach mode for using GDB to debug the test case
2. For a tester who wants to send a failed test case to another developer for additional debugging. It would be nice to be able to just zip/tar just the failed test case test output directory and allow the extracted directory to easily run the failed test case again. The results in this case still might not be the same if the sender and reciever do not have the same QEMU. Though when the QMEU versions are different, I expect the emulated execution on sender and reciever to be close enough to produce the same execution path (maybe not with super old QEMUs, but I don't think its something to be concerned about).

The first patch is needed to make the other two useful. Without it, I know of no easy way to enable grub-shell's debug mode when run via make check. And if grub-shell is not in debug mode then it will remove most/all of the generated files, which would prevent debugging.

Glenn

Glenn Washburn (3):
tests: Allow turning on shell tracing from environment variables
grub-shell: Create run.sh in working directory for easily running test
again
grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial
QEMU arguments

 tests/util/grub-fs-tester.in |  2 ++
 tests/util/grub-shell.in     | 37 +++++++++++++++++++++++++++++-------
 2 files changed, 32 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 tests: Allow turning on shell tracing from environment variables

This allows turning on shell tracing for grub-shell and grub-fs-tester when its not practical or not possible to use commandline arguments (eg. from make check). Turn on tracing when the envvar is an integer greater than 1, since these can generate a lot of output. Since this change uses the environment variables to set the default value for debug in grub-shell, this allows enabling grub-shell's debug mode which will preserve various generated output files that are helpful for debugging tests.

---
 tests/util/grub-fs-tester.in | 2 ++
 tests/util/grub-shell.in     | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)
tests/util/grub-fs-tester.in +2 -0
index cd63941a5f..c9c2d79f22 100644
@@ -6,6 +6,8 @@ export BLKID_FILE=/dev/null
 # We can't have this set, or filesystem tests will fail.
 unset SOURCE_DATE_EPOCH
 
+[ "${GRUB_TEST_DEFAULT_DEBUG:-0}" -gt 1 ] && set -x
+
 fs="$1"
 
 GRUBFSTEST="@builddir@/grub-fstest"
tests/util/grub-shell.in +4 -1
index f5a6e822fd..60494bcf1d 100644
@@ -216,6 +216,7 @@ esac
 
 timeout=60
 mkimage_extra_arg=
+debug=${GRUB_SHELL_DEFAULT_DEBUG:-$GRUB_TEST_DEFAULT_DEBUG}
 
 # Check the arguments.
 for option in "$@"; do
@@ -235,7 +236,7 @@ for option in "$@"; do
     --no-trim)
 	trim=0 ;;
     --debug)
-        debug=1 ;;
+        debug=$((debug+1)) ;;
     --modules=*)
 	ms=`echo "$option" | sed -e 's/--modules=//' -e 's/,/ /g'`
 	modules="$modules $ms" ;;
@@ -320,6 +321,8 @@ for option in "$@"; do
     esac
 done
 
+[ "${debug:-0}" -gt 1 ] && set -x
+
 if [ "x${source}" = x ] ; then
     tmpfile="$work_directory/testcase.cfg"
     while read REPLY; do
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/3 grub-shell: Create run.sh in working directory for easily running test again

In reply to Glenn Washburn

Now it becomes trivial to re-run a test from the output in its working directory. This also makes it easy to send a reproducible failing test to the mailing list. This has allowed a refactor so that the duplicated code to call QEMU has be condensed (eg. the use of timeout and file descriptor redirection). The run.sh script will pass any arguments given to QEMU. This allows QEMU to be easily started in a state ready for GDB to be attached.

---
 tests/util/grub-shell.in | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
tests/util/grub-shell.in +23 -5
index 60494bcf1d..658485f7ea 100644
@@ -521,8 +521,11 @@ if [ x$boot = xnet ]; then
     cp "${cfgfile}" "$netdir/boot/grub/grub.cfg"
     cp "${source}" "$netdir/boot/grub/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$netdir" $files
-    setup_qemu_logger
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic > "$work_directory/qemu-pipe" || ret=$?
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+qemuopts="${qemuopts}"
+exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
+EOF
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
     grubdir="$rootdir/boot/grub"
@@ -541,11 +544,25 @@ elif [ x$boot = xemu ]; then
     [ -z "$files" ] || copy_extra_files "$rootdir" $files
     roottar="$work_directory/root.tar"
     (cd "$rootdir"; tar cf "$roottar" .)
-    setup_qemu_logger
-    "${builddir}/grub-core/grub-emu" -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" > "$work_directory/qemu-pipe" || ret=$?
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+SDIR=\$(realpath -e \${0%/*})
+exec "$(realpath -e "${builddir}")/grub-core/grub-emu" -m "\$SDIR/${device_map##*/}" --memdisk "\$SDIR/${roottar##*/}" -r memdisk -d "/boot/grub"
+EOF
 else
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+SDIR=\$(realpath -e \${0%/*})
+qemuopts="${qemuopts}"
+cd "\$SDIR"
+exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
+EOF
+fi
+
+if [ -f "$work_directory/run.sh" ]; then
     setup_qemu_logger
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} > "$work_directory/qemu-pipe" || ret=$?
+    chmod +x "$work_directory/run.sh"
+    timeout -s KILL $timeout "$work_directory/run.sh" > "$work_directory/qemu-pipe" || ret=$?
 fi
 
 wait
@@ -565,6 +582,7 @@ fi
 test -n "$debug" || rm -f "${isofile}"
 test -n "$debug" || rm -rf "${rom_directory}"
 test -n "$debug" || rm -f "${tmpfile}" "${cfgfile}" "${goutfile}"
+test -n "$debug" || rm -f "$work_directory/run.sh"
 
 exit $ret
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/3 grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial QEMU arguments

In reply to Glenn Washburn

When re-running a failed test, even the non-standard grub-shell QEMU arguments should be preserved in the run.sh to more precisely replay the failed test run.

---
 tests/util/grub-shell.in | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tests/util/grub-shell.in +5 -3
index 658485f7ea..75f71dc1a2 100644
@@ -74,7 +74,7 @@ exec_show_error () {
 work_directory=${WORKDIR:-`mktemp -d "${TMPDIR:-/tmp}/grub-shell.XXXXXXXXXX"`} || exit 1
 
 . "${builddir}/grub-core/modinfo.sh"
-qemuopts="${GRUB_QEMU_OPTS}"
+qemuopts=
 serial_port=com0
 serial_null=
 halt_cmd=halt
@@ -523,8 +523,9 @@ if [ x$boot = xnet ]; then
     [ -z "$files" ] || copy_extra_files "$netdir" $files
     cat >"$work_directory/run.sh" <<EOF
 #! @BUILD_SHEBANG@
+GRUB_QEMU_OPTS=\${GRUB_QEMU_OPTS:-"$GRUB_QEMU_OPTS"}
 qemuopts="${qemuopts}"
-exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
+exec "${qemu}" \${qemuopts} \${GRUB_QEMU_OPTS} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
 EOF
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
@@ -553,9 +554,10 @@ else
     cat >"$work_directory/run.sh" <<EOF
 #! @BUILD_SHEBANG@
 SDIR=\$(realpath -e \${0%/*})
+GRUB_QEMU_OPTS=\${GRUB_QEMU_OPTS:-"$GRUB_QEMU_OPTS"}
 qemuopts="${qemuopts}"
 cd "\$SDIR"
-exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
+exec "${qemu}" \${qemuopts} \${GRUB_QEMU_OPTS} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
 EOF
 fi
 
Signature · 1 line
2.34.1
#4 lore raw
In reply to Glenn Washburn
Quoted 20 lines · Glenn Washburn

When a non-native (eg. grub-shell/qemu) test fails it can be useful to be able to re-run the grub-shell invocation in at least two circumstances.

1. The tester wants to re-run the grub-shell invocation with extra args so that QEMU starts in GDB attach mode for using GDB to debug the test case
2. For a tester who wants to send a failed test case to another developer for additional debugging. It would be nice to be able to just zip/tar just the failed test case test output directory and allow the extracted directory to easily run the failed test case again. The results in this case still might not be the same if the sender and reciever do not have the same QEMU. Though when the QMEU versions are different, I expect the emulated execution on sender and reciever to be close enough to produce the same execution path (maybe not with super old QEMUs, but I don't think its something to be concerned about).

The first patch is needed to make the other two useful. Without it, I know of no easy way to enable grub-shell's debug mode when run via make check. And if grub-shell is not in debug mode then it will remove most/all of the generated files, which would prevent debugging.

Quoted 8 lines

Glenn

Glenn Washburn (3):
tests: Allow turning on shell tracing from environment variables
grub-shell: Create run.sh in working directory for easily running test
again
grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial
QEMU arguments

For all the patches Reviewed-by: Daniel Kiper <[email protected]>...

Daniel