Add run_tests.sh script and update unittests.cfg
v2 cover Add run_tests.sh script and update unittests.cfg
This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.
It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.
I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the x86_64 one so that they actually get tested).
Kevin Wolf (4):
Add run_tests.sh
x86/unittests.cfg: Add arch for x86_64-only tests x86/unittests.cfg: Add missing test cases x86/unittests.cfg: Create test case groups
run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ x86-run | 9 +++- x86/unittests.cfg | 49 +++++++++++++++++++++- 3 files changed, 178 insertions(+), 3 deletions(-) create mode 100755 run_tests.sh
Signature · 1 line
1.8.1.4
PATCH v2 1/4 Add run_tests.sh
This adds a convenient way to run all tests without having to set up Autotest.
--- run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ x86-run | 9 +++- x86/unittests.cfg | 2 + 3 files changed, 132 insertions(+), 2 deletions(-) create mode 100755 run_tests.sh
run_tests.sh new file +123 -0
@@ -0,0 +1,123 @@ +#!/bin/bash + +testroot=x86 +config=$testroot/unittests.cfg +qemu=${qemu:-qemu-system-x86_64} +verbose=0 + +function run() +{ + local testname="$1" + local groups="$2" + local smp="$3" + local kernel="$4" + local opts="$5" + local arch="$6" + + if [ -z "$testname" ]; then + return + fi + + if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then + return + fi + + if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then + echo "skip $1 ($arch only)" + return + fi + + cmdline="./x86-run $kernel -smp $smp -display none $opts" + if [ $verbose != 0 ]; then + echo $cmdline + fi + + # extra_params in the config file may contain backticks that need to be + # expanded, so use eval to start qemu + eval $cmdline >> test.log + + if [ $? -le 1 ]; then + echo -e "\e[32mPASS\e[0m $1" + else + echo -e "\e[31mFAIL\e[0m $1" + fi +} + +function run_all() +{ + local config="$1" + local testname + local smp + local kernel + local opts + local groups + local arch + + exec {config_fd}<$config + + while read -u $config_fd line; do + if [[ "$line" =~ ^\[(.*)\]$ ]]; then + run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" + testname=${BASH_REMATCH[1]} + smp=1 + kernel="" + opts="" + groups="" + arch="" + elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then + kernel=$testroot/${BASH_REMATCH[1]} + elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then + smp=${BASH_REMATCH[1]} + elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then + opts=${BASH_REMATCH[1]} + elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then + groups=${BASH_REMATCH[1]} + elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then + arch=${BASH_REMATCH[1]} + fi + done + + run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" + + exec {config_fd}<&- +} + +function usage() +{ +cat <<EOF + +Usage: $0 [-g group] [-h] [-v] + + -g: Only execute tests in the given group + -h: Output this help text + -v: Enables verbose mode + +Set the environment variable QEMU=/path/to/qemu-system-x86_64 to allow the +internally used x86-run to pick up the right qemu binary. + +EOF +} + +# As it happens, config.mak is valid shell script code, too :-) +source config.mak + +echo > test.log +while getopts "g:hv" opt; do + case $opt in + g) + only_group=$OPTARG + ;; + h) + usage + exit + ;; + v) + verbose=1 + ;; + *) + exit + ;; + esac +done + +run_all $config
x86-run +7 -2
@@ -13,7 +13,7 @@ else qemu="${qemusystem}" else echo QEMU binary ${QEMU} has no support for test device. Exiting. - exit 1 + exit 2 fi fi
@@ -24,4 +24,9 @@ then else command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel" fi -exec ${command} "$@" + +echo ${command} "$@" +${command} "$@" +ret=$? +echo Return value from qemu: $ret +exit $ret
x86/unittests.cfg +2 -0
@@ -3,6 +3,8 @@ # file = foo.flat # Name of the flat file to be used # smp = 2 # Number of processors the VM will use during this test # extra_params = -cpu qemu64,+x2apic # Additional parameters used +# arch = i386/x86_64 # Only if the test case works only on one of them +# groups = group1 group2 # Used to identify test cases with run_tests -g ... [apic] file = apic.flat
Signature · 1 line
1.8.1.4
Re: [PATCH kvm-unittests v2 1/4] Add run_tests.sh
Quoted 85 lines · Kevin Wolf
This adds a convenient way to run all tests without having to set up Autotest.
Signed-off-by Kevin Wolf <[email protected]>--- run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ x86-run | 9 +++- x86/unittests.cfg | 2 + 3 files changed, 132 insertions(+), 2 deletions(-) create mode 100755 run_tests.shrun_tests.sh new file +68
-0@@ -0,0 +1,123 @@ +#!/bin/bash + +testroot=x86 +config=$testroot/unittests.cfg +qemu=${qemu:-qemu-system-x86_64} +verbose=0 + +function run() +{ + local testname="$1" + local groups="$2" + local smp="$3" + local kernel="$4" + local opts="$5" + local arch="$6" + + if [ -z "$testname" ]; then + return + fi + + if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then + return + fi + + if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then + echo "skip $1 ($arch only)" + return + fi + + cmdline="./x86-run $kernel -smp $smp -display none $opts" + if [ $verbose != 0 ]; then + echo $cmdline + fi + + # extra_params in the config file may contain backticks that need to be + # expanded, so use eval to start qemu + eval $cmdline >> test.log + + if [ $? -le 1 ]; then + echo -e "\e[32mPASS\e[0m $1" + else + echo -e "\e[31mFAIL\e[0m $1" + fi +} + +function run_all() +{ + local config="$1" + local testname + local smp + local kernel + local opts + local groups + local arch + + exec {config_fd}<$config + + while read -u $config_fd line; do + if [[ "$line" =~ ^\[(.*)\]$ ]]; then + run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" + testname=${BASH_REMATCH[1]} + smp=1 + kernel="" + opts="" + groups="" + arch="" + elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then + kernel=$testroot/${BASH_REMATCH[1]}
run_tests.sh line 78
Quoted 10 lines
run_tests.sh new file +10
-0+ elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then + smp=${BASH_REMATCH[1]} + elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then + opts=${BASH_REMATCH[1]} + elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then + groups=${BASH_REMATCH[1]} + elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then + arch=${BASH_REMATCH[1]} + fi + done
^ This looks good to me, although using python and the ConfigParser library would be less work (no need to explicitly use regexp based parsing).
I'm currently taking a look at the new fields and how autotest could make use of them... Thanks!
PATCH v2 2/4 x86/unittests.cfg: Add arch for x86_64-only tests
In reply to Kevin WolfTheir kernel binaries would be missing when the tests are built for i386.
--- x86/unittests.cfg | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
x86/unittests.cfg +10 -1
@@ -10,6 +10,7 @@ file = apic.flat smp = 2 extra_params = -cpu qemu64,+x2apic +arch = x86_64 [smptest] file = smptest.flat
@@ -55,15 +56,18 @@ extra_params = -append 'ple_round_robin' [access] file = access.flat +arch = x86_64 [emulator] file = emulator.flat +arch = x86_64 [hypercall] file = hypercall.flat [idt_test] file = idt_test.flat +arch = x86_64 [msr] file = msr.flat
@@ -82,19 +86,23 @@ file = tsc.flat [xsave] file = xsave.flat +arch = x86_64 [rmap_chain] file = rmap_chain.flat +arch = x86_64 [svm] file = svm.flat smp = 2 extra_params = -cpu qemu64,+svm +arch = x86_64 [svm-disabled] file = svm.flat smp = 2 extra_params = -cpu qemu64,-svm +arch = x86_64 [kvmclock_test] file = kvmclock_test.flat
@@ -103,4 +111,5 @@ extra_params = --append "10000000 `date +%s`" [pcid] file = pcid.flat -extra_params = -cpu qemu64,+pcid +extra_params = -cpu qemu64,+pcid +arch = x86_64
Signature · 1 line
1.8.1.4
PATCH v2 3/4 x86/unittests.cfg: Add missing test cases
In reply to Kevin WolfSome test cases seem to have been added without updating the configuration file. This adds them, and leaves cases commented out that don't seem to complete.
--- x86/unittests.cfg | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
x86/unittests.cfg +26 -0
@@ -58,10 +58,16 @@ extra_params = -append 'ple_round_robin' file = access.flat arch = x86_64 +#[asyncpf] +#file = asyncpf.flat + [emulator] file = emulator.flat arch = x86_64 +[eventinj] +file = eventinj.flat + [hypercall] file = hypercall.flat
@@ -69,21 +75,33 @@ file = hypercall.flat file = idt_test.flat arch = x86_64 +#[init] +#file = init.flat + [msr] file = msr.flat +[pmu] +file = pmu.flat + [port80] file = port80.flat [realmode] file = realmode.flat +[s3] +file = s3.flat + [sieve] file = sieve.flat [tsc] file = tsc.flat +[tsc_adjust] +file = tsc_adjust.flat + [xsave] file = xsave.flat arch = x86_64
@@ -104,6 +122,14 @@ smp = 2 extra_params = -cpu qemu64,-svm arch = x86_64 +[taskswitch] +file = taskswitch.flat +arch = i386 + +[taskswitch2] +file = taskswitch2.flat +arch = i386 + [kvmclock_test] file = kvmclock_test.flat smp = 2
Signature · 1 line
1.8.1.4
PATCH v2 4/4 x86/unittests.cfg: Create test case groups
In reply to Kevin WolfPut all vmexit test cases and all task switch test cases into a group, so that you can use something like ./run_tests -g tasks
--- x86/unittests.cfg | 10 ++++++++++ 1 file changed, 10 insertions(+)
x86/unittests.cfg +10 -0
@@ -23,36 +23,44 @@ smp = 3 [vmexit_cpuid] file = vmexit.flat extra_params = -append 'cpuid' +groups = vmexit [vmexit_vmcall] file = vmexit.flat extra_params = -append 'vmcall' +groups = vmexit [vmexit_mov_from_cr8] file = vmexit.flat extra_params = -append 'mov_from_cr8' +groups = vmexit [vmexit_mov_to_cr8] file = vmexit.flat extra_params = -append 'mov_to_cr8' +groups = vmexit [vmexit_inl_pmtimer] file = vmexit.flat extra_params = -append 'inl_from_pmtimer' +groups = vmexit [vmexit_ipi] file = vmexit.flat smp = 2 extra_params = -append 'ipi' +groups = vmexit [vmexit_ipi_halt] file = vmexit.flat smp = 2 extra_params = -append 'ipi_halt' +groups = vmexit [vmexit_ple_round_robin] file = vmexit.flat extra_params = -append 'ple_round_robin' +groups = vmexit [access] file = access.flat
@@ -125,10 +133,12 @@ arch = x86_64 [taskswitch] file = taskswitch.flat arch = i386 +groups = tasks [taskswitch2] file = taskswitch2.flat arch = i386 +groups = tasks [kvmclock_test] file = kvmclock_test.flat
Signature · 1 line
1.8.1.4
Quoted 8 lines · Kevin Wolf
This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.
It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.
Quoted 6 lines
I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the
x86_64one so that they actually get tested).
This has some overlap with a series I sent a few weeks back, enhancing x86-run to read unittests.cfg, and rewriting it in python for native cfg parsing and, cli handling, etc:
http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.html
Thread went silent though...
- Cole
Am 14.04.2013 um 01:26 hat Cole Robinson geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 8 lines
This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.
It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.
Quoted 6 lines
I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the
x86_64one so that they actually get tested).
Quoted 8 lines
This has some overlap with a series I sent a few weeks back, enhancing x86-run to read unittests.cfg, and rewriting it in python for native cfg parsing and, cli handling, etc:
http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.htmlThread went silent though...
I don't really mind which patch is applied as long as I have an easy way to run my tests (with my scripts, ./run_tests.sh -g tasks) and get simple results (one line per testcase, indicating PASS or FAIL) as long as I'm not debugging.
The problem that I see with your patch (5/5 is the interesting one, right?) is that you mix up the kernel file name and the test name; I don't think you can use it with something like the vmexit tests for which several unittests.cfg sections with different options exist. And of course, it's missing some features in comparison.
Is there some feature your Python code has that I don't have yet? I can try to implement it then.
Kevin
Marcelo Tosatti
AppliedQuoted 20 lines · Kevin Wolf
This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.
It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.
I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the
x86_64one so that they actually get tested).Kevin Wolf (4):
Addrun_tests.sh
x86/unittests.cfg: Add arch for x86_64-only tests x86/unittests.cfg: Add missing test cases x86/unittests.cfg: Create test case groups
Quoted 5 lines
run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ x86-run | 9 +++- x86/unittests.cfg | 49 +++++++++++++++++++++- 3 files changed, 178 insertions(+), 3 deletions(-) create mode 100755 run_tests.sh
Applied, thanks.