Skip to contentSkip to conversation

Add run_tests.sh script and update unittests.cfg

PATCH Applied Kevin Wolf 4 participants 9 messages Apr 12, 2013 – Apr 17, 2013 KVM

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

v2 cover Add run_tests.sh script and update unittests.cfg

This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.

It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.

I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the x86_64 one so that they actually get tested).

Kevin Wolf (4):
Add run_tests.sh
x86/unittests.cfg: Add arch for x86_64-only tests x86/unittests.cfg: Add missing test cases x86/unittests.cfg: Create test case groups

 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |  49 +++++++++++++++++++++-
 3 files changed, 178 insertions(+), 3 deletions(-)
 create mode 100755 run_tests.sh
Signature · 1 line
1.8.1.4
author
#1 lore raw

PATCH v2 1/4 Add run_tests.sh

This adds a convenient way to run all tests without having to set up Autotest.

---
 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |   2 +
 3 files changed, 132 insertions(+), 2 deletions(-)
 create mode 100755 run_tests.sh
run_tests.sh new file +123 -0
new file mode 100755
index 0000000..55ecac5
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+testroot=x86
+config=$testroot/unittests.cfg
+qemu=${qemu:-qemu-system-x86_64}
+verbose=0
+
+function run()
+{
+    local testname="$1"
+    local groups="$2"
+    local smp="$3"
+    local kernel="$4"
+    local opts="$5"
+    local arch="$6"
+
+    if [ -z "$testname" ]; then
+        return
+    fi
+
+    if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
+        return
+    fi
+
+    if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
+        echo "skip $1 ($arch only)"
+        return
+    fi
+
+	cmdline="./x86-run $kernel -smp $smp -display none $opts"
+    if [ $verbose != 0 ]; then
+        echo $cmdline
+    fi
+
+    # extra_params in the config file may contain backticks that need to be
+    # expanded, so use eval to start qemu
+    eval $cmdline >> test.log
+
+    if [ $? -le 1 ]; then
+        echo -e "\e[32mPASS\e[0m $1"
+    else
+        echo -e "\e[31mFAIL\e[0m $1"
+    fi
+}
+
+function run_all()
+{
+    local config="$1"
+    local testname
+    local smp
+    local kernel
+    local opts
+    local groups
+    local arch
+
+    exec {config_fd}<$config
+
+    while read -u $config_fd line; do
+        if [[ "$line" =~ ^\[(.*)\]$ ]]; then
+            run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+            testname=${BASH_REMATCH[1]}
+            smp=1
+            kernel=""
+            opts=""
+            groups=""
+            arch=""
+        elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
+            kernel=$testroot/${BASH_REMATCH[1]}
+        elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
+            smp=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
+            opts=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
+            groups=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
+            arch=${BASH_REMATCH[1]}
+        fi
+    done
+
+    run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+
+    exec {config_fd}<&-
+}
+
+function usage()
+{
+cat <<EOF
+
+Usage: $0 [-g group] [-h] [-v]
+
+    -g: Only execute tests in the given group
+    -h: Output this help text
+    -v: Enables verbose mode
+
+Set the environment variable QEMU=/path/to/qemu-system-x86_64 to allow the
+internally used x86-run to pick up the right qemu binary.
+
+EOF
+}
+
+# As it happens, config.mak is valid shell script code, too :-)
+source config.mak
+
+echo > test.log
+while getopts "g:hv" opt; do
+    case $opt in
+        g)
+            only_group=$OPTARG
+            ;;
+        h)
+            usage
+            exit
+            ;;
+        v)
+            verbose=1
+            ;;
+        *)
+            exit
+            ;;
+    esac
+done
+
+run_all $config
x86-run +7 -2
index e395a70..19e5dec 100755
@@ -13,7 +13,7 @@ else
 		qemu="${qemusystem}"
 	else
 		echo QEMU binary ${QEMU} has no support for test device. Exiting.
-		exit 1
+		exit 2
 	fi
 fi
 
@@ -24,4 +24,9 @@ then
 else
 	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel"
 fi
-exec ${command} "$@"
+
+echo ${command} "$@"
+${command} "$@"
+ret=$?
+echo Return value from qemu: $ret
+exit $ret
x86/unittests.cfg +2 -0
index 5e08c55..7d0fa73 100644
@@ -3,6 +3,8 @@
 # file = foo.flat # Name of the flat file to be used
 # smp = 2 # Number of processors the VM will use during this test
 # extra_params = -cpu qemu64,+x2apic # Additional parameters used
+# arch = i386/x86_64 # Only if the test case works only on one of them
+# groups = group1 group2 # Used to identify test cases with run_tests -g ...
 
 [apic]
 file = apic.flat
Signature · 1 line
1.8.1.4
#2 lore raw

Re: [PATCH kvm-unittests v2 1/4] Add run_tests.sh

Quoted 85 lines · Kevin Wolf

This adds a convenient way to run all tests without having to set up Autotest.

---
 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |   2 +
 3 files changed, 132 insertions(+), 2 deletions(-)
 create mode 100755 run_tests.sh
run_tests.sh new file +68 -0
new file mode 100755
index 0000000..55ecac5
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+testroot=x86
+config=$testroot/unittests.cfg
+qemu=${qemu:-qemu-system-x86_64}
+verbose=0
+
+function run()
+{
+    local testname="$1"
+    local groups="$2"
+    local smp="$3"
+    local kernel="$4"
+    local opts="$5"
+    local arch="$6"
+
+    if [ -z "$testname" ]; then
+        return
+    fi
+
+    if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
+        return
+    fi
+
+    if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
+        echo "skip $1 ($arch only)"
+        return
+    fi
+
+       cmdline="./x86-run $kernel -smp $smp -display none $opts"
+    if [ $verbose != 0 ]; then
+        echo $cmdline
+    fi
+
+    # extra_params in the config file may contain backticks that need to be
+    # expanded, so use eval to start qemu
+    eval $cmdline >> test.log
+
+    if [ $? -le 1 ]; then
+        echo -e "\e[32mPASS\e[0m $1"
+    else
+        echo -e "\e[31mFAIL\e[0m $1"
+    fi
+}
+
+function run_all()
+{
+    local config="$1"
+    local testname
+    local smp
+    local kernel
+    local opts
+    local groups
+    local arch
+
+    exec {config_fd}<$config
+
+    while read -u $config_fd line; do
+        if [[ "$line" =~ ^\[(.*)\]$ ]]; then
+            run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+            testname=${BASH_REMATCH[1]}
+            smp=1
+            kernel=""
+            opts=""
+            groups=""
+            arch=""
+        elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
+            kernel=$testroot/${BASH_REMATCH[1]}

run_tests.sh line 78

Quoted 10 lines
run_tests.sh new file +10 -0
+        elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
+            smp=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
+            opts=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
+            groups=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
+            arch=${BASH_REMATCH[1]}
+        fi
+    done

^ This looks good to me, although using python and the ConfigParser library would be less work (no need to explicitly use regexp based parsing).

I'm currently taking a look at the new fields and how autotest could make use of them... Thanks!

author
#3 lore raw

PATCH v2 2/4 x86/unittests.cfg: Add arch for x86_64-only tests

In reply to Kevin Wolf

Their kernel binaries would be missing when the tests are built for i386.

---
 x86/unittests.cfg | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
x86/unittests.cfg +10 -1
index 7d0fa73..f2336bb 100644
@@ -10,6 +10,7 @@
 file = apic.flat
 smp = 2
 extra_params = -cpu qemu64,+x2apic
+arch = x86_64
 
 [smptest]
 file = smptest.flat
@@ -55,15 +56,18 @@ extra_params = -append 'ple_round_robin'
 
 [access]
 file = access.flat
+arch = x86_64
 
 [emulator]
 file = emulator.flat
+arch = x86_64
 
 [hypercall]
 file = hypercall.flat
 
 [idt_test]
 file = idt_test.flat
+arch = x86_64
 
 [msr]
 file = msr.flat
@@ -82,19 +86,23 @@ file = tsc.flat
 
 [xsave]
 file = xsave.flat
+arch = x86_64
 
 [rmap_chain]
 file = rmap_chain.flat
+arch = x86_64
 
 [svm]
 file = svm.flat
 smp = 2
 extra_params = -cpu qemu64,+svm
+arch = x86_64
 
 [svm-disabled]
 file = svm.flat
 smp = 2
 extra_params = -cpu qemu64,-svm
+arch = x86_64
 
 [kvmclock_test]
 file = kvmclock_test.flat
@@ -103,4 +111,5 @@ extra_params = --append "10000000 `date +%s`"
 
 [pcid]
 file = pcid.flat
-extra_params = -cpu qemu64,+pcid
\ No newline at end of file
+extra_params = -cpu qemu64,+pcid
+arch = x86_64
Signature · 1 line
1.8.1.4
author
#4 lore raw

PATCH v2 3/4 x86/unittests.cfg: Add missing test cases

In reply to Kevin Wolf

Some test cases seem to have been added without updating the configuration file. This adds them, and leaves cases commented out that don't seem to complete.

---
 x86/unittests.cfg | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
x86/unittests.cfg +26 -0
index f2336bb..11e8077 100644
@@ -58,10 +58,16 @@ extra_params = -append 'ple_round_robin'
 file = access.flat
 arch = x86_64
 
+#[asyncpf]
+#file = asyncpf.flat
+
 [emulator]
 file = emulator.flat
 arch = x86_64
 
+[eventinj]
+file = eventinj.flat
+
 [hypercall]
 file = hypercall.flat
 
@@ -69,21 +75,33 @@ file = hypercall.flat
 file = idt_test.flat
 arch = x86_64
 
+#[init]
+#file = init.flat
+
 [msr]
 file = msr.flat
 
+[pmu]
+file = pmu.flat
+
 [port80]
 file = port80.flat
 
 [realmode]
 file = realmode.flat
 
+[s3]
+file = s3.flat
+
 [sieve]
 file = sieve.flat
 
 [tsc]
 file = tsc.flat
 
+[tsc_adjust]
+file = tsc_adjust.flat
+
 [xsave]
 file = xsave.flat
 arch = x86_64
@@ -104,6 +122,14 @@ smp = 2
 extra_params = -cpu qemu64,-svm
 arch = x86_64
 
+[taskswitch]
+file = taskswitch.flat
+arch = i386
+
+[taskswitch2]
+file = taskswitch2.flat
+arch = i386
+
 [kvmclock_test]
 file = kvmclock_test.flat
 smp = 2
Signature · 1 line
1.8.1.4
author
#5 lore raw

PATCH v2 4/4 x86/unittests.cfg: Create test case groups

In reply to Kevin Wolf

Put all vmexit test cases and all task switch test cases into a group, so that you can use something like ./run_tests -g tasks

---
 x86/unittests.cfg | 10 ++++++++++
 1 file changed, 10 insertions(+)
x86/unittests.cfg +10 -0
index 11e8077..bc9643e 100644
@@ -23,36 +23,44 @@ smp = 3
 [vmexit_cpuid]
 file = vmexit.flat
 extra_params = -append 'cpuid'
+groups = vmexit
 
 [vmexit_vmcall]
 file = vmexit.flat
 extra_params = -append 'vmcall'
+groups = vmexit
 
 [vmexit_mov_from_cr8]
 file = vmexit.flat
 extra_params = -append 'mov_from_cr8'
+groups = vmexit
 
 [vmexit_mov_to_cr8]
 file = vmexit.flat
 extra_params = -append 'mov_to_cr8'
+groups = vmexit
 
 [vmexit_inl_pmtimer]
 file = vmexit.flat
 extra_params = -append 'inl_from_pmtimer'
+groups = vmexit
 
 [vmexit_ipi]
 file = vmexit.flat
 smp = 2
 extra_params = -append 'ipi'
+groups = vmexit
 
 [vmexit_ipi_halt]
 file = vmexit.flat
 smp = 2
 extra_params = -append 'ipi_halt'
+groups = vmexit
 
 [vmexit_ple_round_robin]
 file = vmexit.flat
 extra_params = -append 'ple_round_robin'
+groups = vmexit
 
 [access]
 file = access.flat
@@ -125,10 +133,12 @@ arch = x86_64
 [taskswitch]
 file = taskswitch.flat
 arch = i386
+groups = tasks
 
 [taskswitch2]
 file = taskswitch2.flat
 arch = i386
+groups = tasks
 
 [kvmclock_test]
 file = kvmclock_test.flat
Signature · 1 line
1.8.1.4
#6 lore raw
In reply to Kevin Wolf
Quoted 8 lines · Kevin Wolf

This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.

It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.

Quoted 6 lines

I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the x86_64 one so that they actually get tested).

This has some overlap with a series I sent a few weeks back, enhancing x86-run to read unittests.cfg, and rewriting it in python for native cfg parsing and, cli handling, etc:

http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.html

Thread went silent though...

- Cole

author
#7 lore raw

Am 14.04.2013 um 01:26 hat Cole Robinson geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 8 lines

This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.

It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.

Quoted 6 lines

I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the x86_64 one so that they actually get tested).

Quoted 8 lines

This has some overlap with a series I sent a few weeks back, enhancing x86-run to read unittests.cfg, and rewriting it in python for native cfg parsing and, cli handling, etc:

http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.html

Thread went silent though...

I don't really mind which patch is applied as long as I have an easy way to run my tests (with my scripts, ./run_tests.sh -g tasks) and get simple results (one line per testcase, indicating PASS or FAIL) as long as I'm not debugging.

The problem that I see with your patch (5/5 is the interesting one, right?) is that you mix up the kernel file name and the test name; I don't think you can use it with something like the vmexit tests for which several unittests.cfg sections with different options exist. And of course, it's missing some features in comparison.

Is there some feature your Python code has that I don't have yet? I can try to implement it then.

Kevin

#8 lore raw
In reply to Kevin Wolf
Quoted 20 lines · Kevin Wolf

This adds a small script that allows to conveniently run all test cases and that reports back one "PASS" or "FAIL" line for each test case; it also creates a test.log file with the full output.

It parses the unittests.cfg file used by autotest, so I'm also making some updates to this file and hopefully make some improvements that autotest can make use of as well.

I checked with Lucas that adding new keys isn't a problem. One thing that is likely to cause failures for autotext, though, is that I'm including i386-only test cases in unittests.cfg. It should probably parse the 'arch' key in the future and skip them (and possibly introduce an i386 run besides the x86_64 one so that they actually get tested).

Kevin Wolf (4):
Add run_tests.sh
x86/unittests.cfg: Add arch for x86_64-only tests x86/unittests.cfg: Add missing test cases x86/unittests.cfg: Create test case groups

Quoted 5 lines
 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |  49 +++++++++++++++++++++-
 3 files changed, 178 insertions(+), 3 deletions(-)
 create mode 100755 run_tests.sh

Applied, thanks.