runtime: Allow additional VMM parameters to be provided
cover runtime: Allow additional VMM parameters to be provided
Occasionally users want to temporarily add additional VMM (QEMU) parameters to all tests (run_tests.sh) or to tests executed with $ARCH/run. With $ARCH/run it's already easy to do, just append them to the command line, but with run_tests.sh there wasn't any way to do it. This series provides two ways. It allows the parameters to be appended to the $QEMU environment variable, which is reasonable thing to expect to work, but doesn't solve the problem of providing parameters that override what is specified in the unittests.cfg file. So VMM_PARAMS is also introduced, which is another environment variable just for the additional parameters, and that variable takes care to show up on the command line in the appropriate places, depending on how the tests are being invoked.
Thanks,
drew
Andrew Jones (2):
arch-run: Allow $QEMU to include parameters
runtime: Introduce VMM_PARAMS
README.md | 5 +++++ arm/run | 4 +++- powerpc/run | 4 +++- s390x/run | 1 + scripts/arch-run.bash | 14 +++++++++++++- scripts/runtime.bash | 4 +++- x86/run | 4 +++- 7 files changed, 31 insertions(+), 5 deletions(-)
Signature · 1 line
2.21.1
PATCH 1/2 arch-run: Allow $QEMU to include parameters
Now it's possible to run all tests using run_tests.sh with a QEMU specified by the $QEMU environment variable that also includes additional parameters. E.g. QEMU="/path/to/qemu -icount 8"
--- scripts/arch-run.bash | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
scripts/arch-run.bash +13 -1
@@ -172,8 +172,20 @@ search_qemu_binary () local save_path=$PATH local qemucmd qemu + if [ -n "$QEMU" ]; then + set -- $QEMU + if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then + echo "\$QEMU environment variable not set to a QEMU binary." >&2 + return 2 + fi + qemu=$(command -v "$1") + shift + echo "$qemu $@" + return + fi + export PATH=$PATH:/usr/libexec - for qemucmd in ${QEMU:-qemu-system-$ARCH_NAME qemu-kvm}; do + for qemucmd in qemu-system-$ARCH_NAME qemu-kvm; do if $qemucmd --help 2>/dev/null | grep -q 'QEMU'; then qemu="$qemucmd" break
Signature · 1 line
2.21.1
Paolo Bonzini
maintainerRe: [PATCH kvm-unit-tests 1/2] arch-run: Allow $QEMU to include parameters
scripts/arch-run.bash search_qemu_binary () · line 183
Quoted 9 lines · Andrew Jones
scripts/arch-run.bash +9
-0+ if [ -n "$QEMU" ]; then + set -- $QEMU + if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then + echo "\$QEMU environment variable not set to a QEMU binary." >&2 + return 2 + fi + qemu=$(command -v "$1") + shift + echo "$qemu $@"
I think $* is more appropriate here. Something like "foo $@ bar" has a weird effect:
$ set -x $ set a b c
$ echo "foo $@ bar" + echo 'foo a' b 'c bar' foo a b c bar
$ echo "foo $* bar" + echo 'foo a b c bar' foo a b c bar
Otherwise, this is a good idea.
Thanks,
Paolo
Quoted 3 lines
scripts/arch-run.bash +3
-0+ return + fi +
Re: [PATCH kvm-unit-tests 1/2] arch-run: Allow $QEMU to include parameters
Quoted 13 lines · Paolo Bonzini
scripts/arch-run.bash search_qemu_binary () · line 183
Quoted 9 lines · Andrew Jones
scripts/arch-run.bash +9
-0+ if [ -n "$QEMU" ]; then + set -- $QEMU + if ! "$1" --help 2>/dev/null | grep -q 'QEMU'; then + echo "\$QEMU environment variable not set to a QEMU binary." >&2 + return 2 + fi + qemu=$(command -v "$1") + shift + echo "$qemu $@"I think $* is more appropriate here. Something like "foo $@ bar" has a weird effect:
Will fix for v2.
Quoted 8 lines
$ set -x $ set a b c$ echo "foo $@ bar" + echo 'foo a' b 'c bar' foo a b c bar
Quoted 5 lines
$ echo "foo $* bar" + echo 'foo a b c bar' foo a b c barOtherwise, this is a good idea.
Thanks,
drew
Quoted 9 lines
Thanks,
Paolo
Quoted 3 lines
scripts/arch-run.bash +3
-0+ return + fi +
PATCH 2/2 runtime: Introduce VMM_PARAMS
In reply to Andrew JonesUsers may need to temporarily provide additional VMM parameters. The VMM_PARAMS environment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that the VMM_PARAMS parameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like $ARCH/run $TEST $PARAMS we want $PARAMS to override $VMM_PARAMS, so we ensure that too.
Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.
--- README.md | 5 +++++ arm/run | 4 +++- powerpc/run | 4 +++- s390x/run | 1 + scripts/runtime.bash | 4 +++- x86/run | 4 +++- 6 files changed, 18 insertions(+), 4 deletions(-)
README.md +5 -0
@@ -47,6 +47,11 @@ environment variable: QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat +If QEMU additional parameters are needed for all tests, then they may be +provided in the VMM_PARAMS environment variable: + + VMM_PARAMS="-additional parameters -go here" ./run_tests.sh + To select an accelerator, for example "kvm" or "tcg", specify the ACCEL=name environment variable:
arm/run +3 -1
@@ -60,7 +60,9 @@ fi M+=",accel=$ACCEL" command="$qemu -nodefaults $M -cpu $processor $chr_testdev $pci_testdev" -command+=" -display none -serial stdio -kernel" +command+=" -display none -serial stdio" +command+=" $VMM_PARAMS" +command+=" -kernel" command="$(timeout_cmd) $command" run_qemu $command "$@"
powerpc/run +3 -1
@@ -23,7 +23,9 @@ fi M='-machine pseries' M+=",accel=$ACCEL" command="$qemu -nodefaults $M -bios $FIRMWARE" -command+=" -display none -serial stdio -kernel" +command+=" -display none -serial stdio" +command+=" $VMM_PARAMS" +command+=" -kernel" command="$(migration_cmd) $(timeout_cmd) $command" # powerpc tests currently exit with rtas-poweroff, which exits with 0.
s390x/run +1 -0
@@ -19,6 +19,7 @@ M='-machine s390-ccw-virtio' M+=",accel=$ACCEL" command="$qemu -nodefaults -nographic $M" command+=" -chardev stdio,id=con0 -device sclpconsole,chardev=con0" +command+=" $VMM_PARAMS" command+=" -kernel" command="$(timeout_cmd) $command"
scripts/runtime.bash +3 -1
@@ -30,7 +30,9 @@ premature_failure() get_cmdline() { local kernel=$1 - echo "TESTNAME=$testname TIMEOUT=$timeout ACCEL=$accel $RUNTIME_arch_run $kernel -smp $smp $opts" + + # Move VMM_PARAMS to the end to override parameters provided in unittests.cfg:extra_params + echo "VMM_PARAMS= TESTNAME=$testname TIMEOUT=$timeout ACCEL=$accel $RUNTIME_arch_run $kernel -smp $smp $opts $VMM_PARAMS" } skip_nodefault()
x86/run +3 -1
@@ -38,7 +38,9 @@ else fi command="${qemu} -nodefaults $pc_testdev -vnc none -serial stdio $pci_testdev" -command+=" -machine accel=$ACCEL -kernel" +command+=" -machine accel=$ACCEL" +command+=" $VMM_PARAMS" +command+=" -kernel" command="$(timeout_cmd) $command" run_qemu ${command} "$@"
Signature · 1 line
2.21.1
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 7 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.
While reading this, I was wondering why not simply "$QEMU_PARAMS"?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.While reading this, I was wondering why not simply "$QEMU_PARAMS"?
I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.
Thanks,
drew
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 12 lines · Andrew Jones
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.While reading this, I was wondering why not simply "$QEMU_PARAMS"?
Quoted 9 lines
I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.
And we do expect other VMMs to eat the same set of parameters? If it's QEMU specific, I think we should make this clear.
But no strong opinion.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 23 lines · David Hildenbrand
Quoted 12 lines · Andrew Jones
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.While reading this, I was wondering why not simply "$QEMU_PARAMS"?
Quoted 9 lines
I'd like to slowly move away from assuming QEMU is the VMM. We already have support for kvmtool (to some degree) and I'd like to see that support increase. Also, maybe we'll eventually add support for a rust-vmm reference VMM to drive kvm-unit-tests. IOW, rather than add yet another QEMU named variable I'd like to start a trend of using VMM named variables. Actually, we could add VMM named alternatives for the QEMU named ones we have now and start using them in the scripts. We'd just need to remap the old names for backward compatibility early in the run.
Quoted 2 lines
And we do expect other VMMs to eat the same set of parameters? If it's QEMU specific, I think we should make this clear.
It won't matter. The parameters are VMM specific. Whether or not where they show up on the command line has the same semantics as QEMU could be an issue, but what the parameters are is up to the user, and they should know which vmm they're currently using.
kvmtool support isn't complete because the run scripts haven't yet provided a way for non-QEMU vmms to use the unittests.cfg files. Andre had some patches once, though. So maybe someday.
Thanks,
drew
Paolo Bonzini
maintainerRe: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
In reply to Andrew JonesQuoted 8 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.
Quoted 5 lines
Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.
What about looking for "--" and passing to QEMU all parameters after it?
Paolo
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 9 lines · Paolo Bonzini
Quoted 8 lines · Andrew Jones
Users may need to temporarily provide additional VMM parameters. The
VMM_PARAMSenvironment variable provides a way to do that. We take care to make sure when executing ./run_tests.sh that theVMM_PARAMSparameters come last, allowing unittests.cfg parameters to be overridden. However, when running a command line like$ARCH/run $TEST $PARAMSwe want $PARAMS to override $VMM_PARAMS, so we ensure that too.
Quoted 8 lines
Quoted 5 lines
Additional QEMU parameters can still be provided by appending them to the $QEMU environment variable when it provides the path to QEMU, but as those parameters will then be the first in the command line they cannot override anything, and may themselves be overridden.
What about looking for "--" and passing to QEMU all parameters after it?
That was the way we originally planned on doing it when Alex Bennee posted his patch. However since d4d34e648482 ("run_tests: fix command line options handling") the "--" has become the divider between run_tests.sh parameter inputs and individually specified tests. We'd have to change that behavior, potentially breaking command lines, to go back to the "--" approach.
Also, the nice thing about using an environment variable is that it works with standalone tests too.
VMM_PARAMS="-foo bar" tests/mytest
will run the test with "-foo bar" appended to the command line. We could modify mkstandalone.sh to get that feature too (allowing the additional parameters to be given after tests/mytest), but with VMM_PARAMS we don't have to.
Thanks,
drew
Paolo Bonzini
maintainerRe: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 4 lines · Andrew Jones
That was the way we originally planned on doing it when Alex Bennee posted his patch. However since
d4d34e648482("run_tests: fix command line options handling") the "--" has become the divider betweenrun_tests.shparameter inputs and individually specified tests.
Hmm, more precisely that is how getopt separates options from other parameters.
Since we don't expect test names starting with a dash, we could do something like (untested):
run_tests.sh +14 -1
@@ -35,7 +35,20 @@ RUNTIME_arch_run="./$TEST_DIR/run" source scripts/runtime.bash only_tests="" -args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $*` +args="" +vmm_args="" +while [ $# -gt 0 ]; do + if test "$1" = --; then + shift + vmm_args=$* + break + else + args="args $1" + shift + fi +done + +args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $args` [ $? -ne 0 ] && exit 2; set -- $args; while [ $# -gt 0 ]; do
Quoted 4 lines
will run the test with "-foo bar" appended to the command line. We could modify
mkstandalone.shto get that feature too (allowing the additional parameters to be given after tests/mytest), but withVMM_PARAMSwe don't have to.
Yes, having consistency between standalone and run_tests.sh is a good thing.
Paolo
Re: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 28 lines · Paolo Bonzini
Quoted 4 lines · Andrew Jones
That was the way we originally planned on doing it when Alex Bennee posted his patch. However since
d4d34e648482("run_tests: fix command line options handling") the "--" has become the divider betweenrun_tests.shparameter inputs and individually specified tests.Hmm, more precisely that is how getopt separates options from other parameters.
Since we don't expect test names starting with a dash, we could do something like (untested):
run_tests.sh +7
-1@@ -35,7 +35,20 @@ RUNTIME_arch_run="./$TEST_DIR/run" source scripts/runtime.bash only_tests="" -args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $*` +args="" +vmm_args="" +while [ $# -gt 0 ]; do + if test "$1" = --; then + shift + vmm_args=$* + break
run_tests.sh RUNTIME_arch_run="./$TEST_DIR/run" · line 54
Quoted 10 lines
run_tests.sh +7
-0+ else + args="args $1" + shift + fi +done + +args=`getopt -u -o ag:htj:v -l all,group:,help,tap13,parallel:,verbose -- $args` [ $? -ne 0 ] && exit 2; set -- $args; while [ $# -gt 0 ]; do
Unfortunately it regresses the current command line. Here's what I tested
Before
------
$ ./run_tests.sh -j 2 -v pmu psci VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1 VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP PASS pmu (3 tests) PASS psci (4 tests)
$ ./run_tests.sh pmu psci -j 2 -v VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1 VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP PASS pmu (3 tests) PASS psci (4 tests)
$ ./run_tests.sh -j 2 -v -- pmu psci VMM_PARAMS= TESTNAME=pmu TIMEOUT=90s ACCEL= ./arm/run arm/pmu.flat -smp 1 VMM_PARAMS= TESTNAME=psci TIMEOUT=90s ACCEL= ./arm/run arm/psci.flat -smp $MAX_SMP PASS pmu (3 tests) PASS psci (4 tests)
After
-----
$ ./run_tests.sh -j 2 -v pmu psci PASS psci (4 tests)
$ ./run_tests.sh pmu psci -j 2 -v $
(no output)
$ ./run_tests.sh -j 2 -v -- pmu psci $
(no output)
Quoted 8 lines
Quoted 5 lines
will run the test with "-foo bar" appended to the command line. We could modify
mkstandalone.shto get that feature too (allowing the additional parameters to be given after tests/mytest), but withVMM_PARAMSwe don't have to.Yes, having consistency between standalone and
run_tests.shis a good thing.
So should we:
1) try to get "--" working, but also keep the environment variable as an alternative which works with standalone?
2) drop the environment variable, get "--" working and modify mkstandalone.sh?
3) drop the environment variable, get "--" working, but forget about standalone?
4) just keep the VMM_PARAMS approach and forget about "--"?
Thanks,
drew
Paolo Bonzini
maintainerRe: [PATCH kvm-unit-tests 2/2] runtime: Introduce VMM_PARAMS
Quoted 10 lines · Andrew Jones
After
-----$ ./run_tests.sh -j 2 -v pmu psci PASS psci (4 tests)$ ./run_tests.sh pmu psci -j 2 -v $(no output)
Bugs. :)
Quoted 4 lines
$ ./run_tests.sh -j 2 -v -- pmu psci $(no output)
This is intended, it shouldn't be a big deal because we don't have tests starting with a dash.
Paolo