Skip to contentSkip to conversation

runtime: Allow additional VMM parameters to be provided

PATCH In review Andrew Jones 3 participants 14 messages Feb 13, 2020 – Feb 14, 2020 KVM

lore
Share & tools
lore
#0 lore raw

cover runtime: Allow additional VMM parameters to be provided

Occasionally users want to temporarily add additional VMM (QEMU) parameters to all tests (run_tests.sh) or to tests executed with $ARCH/run. With $ARCH/run it's already easy to do, just append them to the command line, but with run_tests.sh there wasn't any way to do it. This series provides two ways. It allows the parameters to be appended to the $QEMU environment variable, which is reasonable thing to expect to work, but doesn't solve the problem of providing parameters that override what is specified in the unittests.cfg file. So VMM_PARAMS is also introduced, which is another environment variable just for the additional parameters, and that variable takes care to show up on the command line in the appropriate places, depending on how the tests are being invoked.

Thanks,
drew

Andrew Jones (2):
arch-run: Allow $QEMU to include parameters
runtime: Introduce VMM_PARAMS

 README.md             |  5 +++++
 arm/run               |  4 +++-
 powerpc/run           |  4 +++-
 s390x/run             |  1 +
 scripts/arch-run.bash | 14 +++++++++++++-
 scripts/runtime.bash  |  4 +++-
 x86/run               |  4 +++-
 7 files changed, 31 insertions(+), 5 deletions(-)
Signature · 1 line
2.21.1
#1 lore raw

PATCH 1/2 arch-run: Allow $QEMU to include parameters

Now it's possible to run all tests using run_tests.sh with a QEMU specified by the $QEMU environment variable that also includes additional parameters. E.g. QEMU="/path/to/qemu -icount 8"

Cc Peter Maydell <[email protected]>
Cc Alex Bennée <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Laurent Vivier <[email protected]>
Cc Thomas Huth <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Janosch Frank <[email protected]>
---
 scripts/arch-run.bash | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
scripts/arch-run.bash +13 -1
index d3ca19d49952..ebe4d3cb2a09 100644
@@ -172,8 +172,20 @@ search_qemu_binary ()
 	local save_path=$PATH
 	local qemucmd qemu
 
+	if [ -n "$QEMU" ]; then
+		set -- $QEMU
+		if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then
+			echo "\$QEMU environment variable not set to a QEMU binary." >&2
+			return 2
+		fi
+		qemu=$(command -v "$1")
+		shift
+		echo "$qemu $@"
+		return
+	fi
+
 	export PATH=$PATH:/usr/libexec
-	for qemucmd in ${QEMU:-qemu-system-$ARCH_NAME qemu-kvm}; do
+	for qemucmd in qemu-system-$ARCH_NAME qemu-kvm; do
 		if $qemucmd --help 2>/dev/null | grep -q 'QEMU'; then
 			qemu="$qemucmd"
 			break
Signature · 1 line
2.21.1
maintainer
#2 lore raw

Re: [PATCH kvm-unit-tests 1/2] arch-run: Allow $QEMU to include parameters

scripts/arch-run.bash search_qemu_binary () · line 183

Quoted 9 lines · Andrew Jones
scripts/arch-run.bash +9 -0
+	if [ -n "$QEMU" ]; then
+		set -- $QEMU
+		if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then
+			echo "\$QEMU environment variable not set to a QEMU binary." >&2
+			return 2
+		fi
+		qemu=$(command -v "$1")
+		shift
+		echo "$qemu $@"

I think $* is more appropriate here. Something like "foo $@ bar" has a weird effect:

	$ set -x
	$ set a b c
	$ echo "foo $@ bar"
	+ echo 'foo a' b 'c bar'
	foo a b c bar
	$ echo "foo $* bar"
	+ echo 'foo a b c bar'
	foo a b c bar

Otherwise, this is a good idea.

Thanks,

Paolo

Quoted 3 lines
scripts/arch-run.bash +3 -0
+		return
+	fi
+
#3 lore raw

Re: [PATCH kvm-unit-tests 1/2] arch-run: Allow $QEMU to include parameters

Quoted 13 lines · Paolo Bonzini

scripts/arch-run.bash search_qemu_binary () · line 183

Quoted 9 lines · Andrew Jones
scripts/arch-run.bash +9 -0
+	if [ -n "$QEMU" ]; then
+		set -- $QEMU
+		if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then
+			echo "\$QEMU environment variable not set to a QEMU binary." >&2
+			return 2
+		fi
+		qemu=$(command -v "$1")
+		shift
+		echo "$qemu $@"

I think $* is more appropriate here. Something like "foo $@ bar" has a weird effect:

Will fix for v2.

Quoted 8 lines
	$ set -x
	$ set a b c
	$ echo "foo $@ bar"
	+ echo 'foo a' b 'c bar'
	foo a b c bar
Quoted 5 lines
	$ echo "foo $* bar"
	+ echo 'foo a b c bar'
	foo a b c bar

Otherwise, this is a good idea.

Thanks,
drew

Quoted 9 lines

Thanks,

Paolo

Quoted 3 lines
scripts/arch-run.bash +3 -0
+		return
+	fi
+
#4 lore raw

PATCH 2/2 runtime: Introduce VMM_PARAMS

In reply to Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.

Cc Peter Maydell <[email protected]>
Cc Alex Bennée <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Laurent Vivier <[email protected]>
Cc Thomas Huth <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Janosch Frank <[email protected]>
---
 README.md            | 5 +++++
 arm/run              | 4 +++-
 powerpc/run          | 4 +++-
 s390x/run            | 1 +
 scripts/runtime.bash | 4 +++-
 x86/run              | 4 +++-
 6 files changed, 18 insertions(+), 4 deletions(-)
README.md +5 -0
index 396fbf809a4e..834d6202ac97 100644
@@ -47,6 +47,11 @@ environment variable:
 
     QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat
 
+If QEMU additional parameters are needed for all tests, then they may be
+provided in the VMM_PARAMS environment variable:
+
+    VMM_PARAMS="-additional parameters -go here" ./run_tests.sh
+
 To select an accelerator, for example "kvm" or "tcg", specify the
 ACCEL=name environment variable:
 
arm/run +3 -1
index 277db9bb4a02..a8a93591b825 100755
@@ -60,7 +60,9 @@ fi
 
 M+=",accel=$ACCEL"
 command="$qemu -nodefaults $M -cpu $processor $chr_testdev $pci_testdev"
-command+=" -display none -serial stdio -kernel"
+command+=" -display none -serial stdio"
+command+=" $VMM_PARAMS"
+command+=" -kernel"
 command="$(timeout_cmd) $command"
 
 run_qemu $command "$@"
powerpc/run +3 -1
index 597ab96ed8a8..b07aa18f26bf 100755
@@ -23,7 +23,9 @@ fi
 M='-machine pseries'
 M+=",accel=$ACCEL"
 command="$qemu -nodefaults $M -bios $FIRMWARE"
-command+=" -display none -serial stdio -kernel"
+command+=" -display none -serial stdio"
+command+=" $VMM_PARAMS"
+command+=" -kernel"
 command="$(migration_cmd) $(timeout_cmd) $command"
 
 # powerpc tests currently exit with rtas-poweroff, which exits with 0.
s390x/run +1 -0
index 0980504448ce..9bfb95397064 100755
@@ -19,6 +19,7 @@ M='-machine s390-ccw-virtio'
 M+=",accel=$ACCEL"
 command="$qemu -nodefaults -nographic $M"
 command+=" -chardev stdio,id=con0 -device sclpconsole,chardev=con0"
+command+=" $VMM_PARAMS"
 command+=" -kernel"
 command="$(timeout_cmd) $command"
 
scripts/runtime.bash +3 -1
index eb6089091e23..5ea3772d9b2b 100644
@@ -30,7 +30,9 @@ premature_failure()
 get_cmdline()
 {
     local kernel=$1
-    echo "TESTNAME=$testname TIMEOUT=$timeout ACCEL=$accel $RUNTIME_arch_run $kernel -smp $smp $opts"
+
+    # Move VMM_PARAMS to the end to override parameters provided in unittests.cfg:extra_params
+    echo "VMM_PARAMS= TESTNAME=$testname TIMEOUT=$timeout ACCEL=$accel $RUNTIME_arch_run $kernel -smp $smp $opts $VMM_PARAMS"
 }
 
 skip_nodefault()
x86/run +3 -1
index 1ac91f1d880f..3770c16ad4e6 100755
@@ -38,7 +38,9 @@ else
 fi
 
 command="${qemu} -nodefaults $pc_testdev -vnc none -serial stdio $pci_testdev"
-command+=" -machine accel=$ACCEL -kernel"
+command+=" -machine accel=$ACCEL"
+command+=" $VMM_PARAMS"
+command+=" -kernel"
 command="$(timeout_cmd) $command"
 
 run_qemu ${command} "$@"
Signature · 1 line
2.21.1
#5 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 7 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

While reading this, I was wondering why not simply "$QEMU_PARAMS"?

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

While reading this, I was wondering why not simply "$QEMU_PARAMS"?

I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.

Thanks,
drew

#7 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 12 lines · Andrew Jones
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

While reading this, I was wondering why not simply "$QEMU_PARAMS"?

Quoted 9 lines

I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.

And we do expect other VMMs to eat the same set of parameters? If it's QEMU specific, I think we should make this clear.

But no strong opinion.

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 23 lines · David Hildenbrand
Quoted 12 lines · Andrew Jones
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

While reading this, I was wondering why not simply "$QEMU_PARAMS"?

Quoted 9 lines

I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.

Quoted 2 lines

And we do expect other VMMs to eat the same set of parameters? If it's QEMU specific, I think we should make this clear.

It won't matter. The parameters are VMM specific. Whether or not where they show up on the command line has the same semantics as QEMU could be an issue, but what the parameters are is up to the user, and they should know which vmm they're currently using.

kvmtool support isn't complete because the run scripts haven't yet provided a way for non-QEMU vmms to use the unittests.cfg files. Andre had some patches once, though. So maybe someday.

Thanks,
drew

maintainer
#9 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

In reply to Andrew Jones
Quoted 8 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

Quoted 5 lines

Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.

What about looking for "--" and passing to QEMU all parameters after it?

Paolo

#10 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 9 lines · Paolo Bonzini
Quoted 8 lines · Andrew Jones

Users may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.

Quoted 8 lines
Quoted 5 lines

Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.

What about looking for "--" and passing to QEMU all parameters after it?

That was the way we originally planned on doing it when Alex Bennee posted his patch. However since d4d34e648482 ("run_tests: fix command line options handling") the "--" has become the divider between run_tests.sh parameter inputs and individually specified tests. We'd have to change that behavior, potentially breaking command lines, to go back to the "--" approach.

Also, the nice thing about using an environment variable is that it works with standalone tests too.

VMM_PARAMS="-foo bar" tests/mytest

will run the test with "-foo bar" appended to the command line. We could modify mkstandalone.sh to get that feature too (allowing the additional parameters to be given after tests/mytest), but with VMM_PARAMS we don't have to.

Thanks,
drew

maintainer
#11 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 4 lines · Andrew Jones

That was the way we originally planned on doing it when Alex Bennee posted his patch. However since d4d34e648482 ("run_tests: fix command line options handling") the "--" has become the divider between run_tests.sh parameter inputs and individually specified tests.

Hmm, more precisely that is how getopt separates options from other parameters.

Since we don't expect test names starting with a dash, we could do something like (untested):

run_tests.sh +14 -1
index 01e36dc..8b71cf2 100755
@@ -35,7 +35,20 @@ RUNTIME_arch_run="./$TEST_DIR/run"
 source scripts/runtime.bash
 
 only_tests=""
-args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $*`
+args=""
+vmm_args=""
+while [ $# -gt 0 ]; do
+   if test "$1" = --; then
+       shift
+       vmm_args=$*
+       break
+   else
+       args="args $1"
+       shift
+   fi
+done
+
+args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $args`
 [ $? -ne 0 ] && exit 2;
 set -- $args;
 while [ $# -gt 0 ]; do
Quoted 4 lines

will run the test with "-foo bar" appended to the command line. We could modify mkstandalone.sh to get that feature too (allowing the additional parameters to be given after tests/mytest), but with VMM_PARAMS we don't have to.

Yes, having consistency between standalone and run_tests.sh is a good thing.

Paolo

#12 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 28 lines · Paolo Bonzini
Quoted 4 lines · Andrew Jones

That was the way we originally planned on doing it when Alex Bennee posted his patch. However since d4d34e648482 ("run_tests: fix command line options handling") the "--" has become the divider between run_tests.sh parameter inputs and individually specified tests.

Hmm, more precisely that is how getopt separates options from other parameters.

Since we don't expect test names starting with a dash, we could do something like (untested):

run_tests.sh +7 -1
index 01e36dc..8b71cf2 100755
@@ -35,7 +35,20 @@ RUNTIME_arch_run="./$TEST_DIR/run"
 source scripts/runtime.bash
 
 only_tests=""
-args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $*`
+args=""
+vmm_args=""
+while [ $# -gt 0 ]; do
+   if test "$1" = --; then
+       shift
+       vmm_args=$*
+       break

run_tests.sh RUNTIME_arch_run="./$TEST_DIR/run" · line 54

Quoted 10 lines
run_tests.sh +7 -0
+   else
+       args="args $1"
+       shift
+   fi
+done
+
+args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $args`
 [ $? -ne 0 ] && exit 2;
 set -- $args;
 while [ $# -gt 0 ]; do

Unfortunately it regresses the current command line. Here's what I tested

Before
------

$ ./run_tests.sh -j 2 -v pmu psci
VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1
VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP
PASS pmu (3 tests)
PASS psci (4 tests)
$ ./run_tests.sh pmu psci -j 2 -v
VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1
VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP
PASS pmu (3 tests)
PASS psci (4 tests)
$ ./run_tests.sh -j 2 -v -- pmu psci
VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1
VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP
PASS pmu (3 tests)
PASS psci (4 tests)

After
-----

$ ./run_tests.sh -j 2 -v pmu psci
PASS psci (4 tests)
$ ./run_tests.sh pmu psci -j 2 -v
$

(no output)

$ ./run_tests.sh -j 2 -v -- pmu psci
$

(no output)

Quoted 8 lines
Quoted 5 lines

will run the test with "-foo bar" appended to the command line. We could modify mkstandalone.sh to get that feature too (allowing the additional parameters to be given after tests/mytest), but with VMM_PARAMS we don't have to.

Yes, having consistency between standalone and run_tests.sh is a good thing.

So should we:

1) try to get "--" working, but also keep the environment variable as an alternative which works with standalone?
2) drop the environment variable, get "--" working and modify mkstandalone.sh?
3) drop the environment variable, get "--" working, but forget about standalone?
4) just keep the VMM_PARAMS approach and forget about "--"?

Thanks,
drew

maintainer
#13 lore raw

Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS

Quoted 10 lines · Andrew Jones

After
-----

$ ./run_tests.sh -j 2 -v pmu psci
PASS psci (4 tests)
$ ./run_tests.sh pmu psci -j 2 -v
$

(no output)

Bugs. :)

Quoted 4 lines
$ ./run_tests.sh -j 2 -v -- pmu psci
$

(no output)

This is intended, it shouldn't be a big deal because we don't have tests starting with a dash.

Paolo