kvm-unit-test: more documentation and runner script
PATCH 1/1 kvm-unit-test: more documentation and runner script
Add documentation about using qemu-system for unit tests. Add runner script to select the correct binary and flags.
--- README | 10 +++++++++- x86-run | 27 +++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) create mode 100755 x86-run
README +9 -1
@@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory. Tests' images created in ./<ARCH>/*.flat An example of a test invocation: -qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat +Using qemu-kvm: + +qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat This invocation runs the msr test case. The test outputs to stdio. +Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat +Or use a runner script to detect the correct invocation: +./x86-run ./x86/msr.flat +To select a specific qemu binary, specify the QEMU=<path> environment: +QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat Directory structure: .: Makefile and config files for the tests
x86-run new file +27 -0
@@ -0,0 +1,27 @@ +#!/usr/bin/bash + +qemukvm="${QEMU:-qemu-kvm}" +qemusystem="${QEMU:-qemu-system-x86_64}" +if + ${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; +then + qemu="${qemukvm}" +else + if + ${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; + then + qemu="${qemusystem}" + else + echo QEMU binary ${QEMU} has no support for test device. Exiting. + exit 1 + fi +fi + +if + ${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null; +then + command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel" +else + command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat" +fi +exec ${command} "$@"
Signature · 1 line
MST
Quoted 16 lines · Michael S. Tsirkin
Add documentation about using qemu-system for unit tests. Add runner script to select the correct binary and flags.
Signed-off-by Michael S. Tsirkin <[email protected]>--- README | 10 +++++++++- x86-run | 27 +++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-)create mode 100755 x86-run
README +0
-0@@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory. Tests' images created in ./<ARCH>/*.flat
README To create the tests' images just type 'make' in this directory. · line 19
Quoted 10 lines
README +5
-1An example of a test invocation: -qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat +Using qemu-kvm: + +qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat This invocation runs the msr test case. The test outputs to stdio. +Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
I think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
Quoted 30 lines
README +4
-0+Or use a runner script to detect the correct invocation: +./x86-run ./x86/msr.flat +To select a specific qemu binary, specify the QEMU=<path> environment: +QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat Directory structure: .: Makefile and config files for the testsx86-run new file +17
-0@@ -0,0 +1,27 @@ +#!/usr/bin/bash + +qemukvm="${QEMU:-qemu-kvm}" +qemusystem="${QEMU:-qemu-system-x86_64}" +if + ${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; +then + qemu="${qemukvm}" +else + if + ${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; + then + qemu="${qemusystem}" + else + echo QEMU binary ${QEMU} has no support for test device. Exiting. + exit 1 + fi
x86-run line 27
Quoted 10 lines
x86-run new file +10
-0+fi + +if + ${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null; +then + command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel" +else + command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat" +fi +exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
Quoted 67 lines · Lucas Meneghel Rodrigues
Quoted 16 lines · Michael S. Tsirkin
Add documentation about using qemu-system for unit tests. Add runner script to select the correct binary and flags.
Signed-off-by Michael S. Tsirkin <[email protected]>--- README | 10 +++++++++- x86-run | 27 +++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) create mode 100755 x86-runREADME +0
-0@@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory. Tests' images created in ./<ARCH>/*.flatREADME To create the tests' images just type 'make' in this directory. · line 19
Quoted 10 lines
README +5
-1An example of a test invocation: -qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat +Using qemu-kvm: + +qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat This invocation runs the msr test case. The test outputs to stdio. +Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
Quoted 35 lines
README +4
-0+Or use a runner script to detect the correct invocation: +./x86-run ./x86/msr.flat +To select a specific qemu binary, specify the QEMU=<path> environment: +QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat Directory structure: .: Makefile and config files for the testsx86-run new file +22
-0@@ -0,0 +1,27 @@ +#!/usr/bin/bash + +qemukvm="${QEMU:-qemu-kvm}" +qemusystem="${QEMU:-qemu-system-x86_64}" +if + ${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; +then + qemu="${qemukvm}" +else + if + ${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null; + then + qemu="${qemusystem}" + else + echo QEMU binary ${QEMU} has no support for test device. Exiting. + exit 1 + fi +fi + +if + ${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null; +then
Quoted 8 lines
x86-run line 27
Quoted 5 lines
x86-run new file +5
-0+ command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel" +else + command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat" +fi +exec ${command} "$@"^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.
Signature · 1 line
MST
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
README To create the tests' images just type 'make' in this directory. · line 19
Quoted 2 lines
README +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.
See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure. Anyway, it was just a suggestion, feel free to disregard it.
[1] having to parse the output and try to guess what is a pass or fail is a mess at best, and should be avoided unless we positively have no saner way of doing it.
Quoted 16 lines · Lucas Meneghel Rodrigues
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
README To create the tests' images just type 'make' in this directory. · line 19
Quoted 2 lines
README +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
Quoted 8 lines
Quoted 3 lines
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure.
Right but given a qemu binary, how do I find out what it is on success and what it is on failure?
Quoted 6 lines
Anyway, it was just a
suggestion, feel free to disregard it.[1] having to parse the output and try to guess what is a pass or fail is a mess at best, and should be avoided unless we positively have no saner way of doing it.
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Lucas Meneghel Rodrigues
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
README To create the tests' images just type 'make' in this directory. · line 19
Quoted 2 lines
README +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.
Quoted 8 lines
Quoted 4 lines
See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure.
Right but given a qemu binary, how do I find out what it is on success and what it is on failure?
Since you know what device you are using you know expected value for successful/failure.
Quoted 6 lines
Quoted 6 lines
Anyway, it was just a
suggestion, feel free to disregard it.[1] having to parse the output and try to guess what is a pass or fail is a mess at best, and should be avoided unless we positively have no saner way of doing it.
--
Gleb.
Quoted 27 lines · Gleb Natapov
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Lucas Meneghel Rodrigues
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
README To create the tests' images just type 'make' in this directory. · line 19
Quoted 2 lines
README +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure.
Quoted 5 lines
Quoted 3 lines
Right but given a qemu binary, how do I find out what it is on success and what it is on failure?
Since you know what device you are using you know expected value for successful/failure.
So exit status is 1 for success 0 for failure?
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Anyway, it was just a
suggestion, feel free to disregard it.[1] having to parse the output and try to guess what is a pass or fail is a mess at best, and should be avoided unless we positively have no saner way of doing it.
--
Gleb.
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Gleb Natapov
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Lucas Meneghel Rodrigues
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Right but given a qemu binary, how do I find out what it is on success and what it is on failure?
Since you know what device you are using you know expected value for successful/failure.
So exit status is 1 for success 0 for failure?
As Lucas said above upstream is different from qemu-kvm unfortunately. On upstream you need to shift return value right for 1 bit to get to the tests return value (why?) and then 0 will be success otherwise failure.
--
Gleb.
Quoted 29 lines · Gleb Natapov
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Gleb Natapov
Quoted 26 lines · Michael S. Tsirkin
Quoted 24 lines · Lucas Meneghel Rodrigues
Quoted 6 lines · Michael S. Tsirkin
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+Using qemu (supported since qemu 1.3): +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flatI think it is worth here to point out that with new qemu, after the unittest is done, the exit status of qemu is 1, different from the 'old style', whose exit status in successful completion is 0.
^ "comment above"
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
+exec ${command} "$@"
^ What about checking the exit status of qemu here and print something like "test $@ PASS" or "test $@ FAIL"?
How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.See comment above. Well, test output may be good for humans, but it is really not good for machines [1], that's why when the test suite was developed, the convention was to make qemu to exit with a given return code on success and others on failure.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Right but given a qemu binary, how do I find out what it is on success and what it is on failure?
Since you know what device you are using you know expected value for successful/failure.
So exit status is 1 for success 0 for failure?
For qemu-kvm, 0 success, non 0 for failure
For qemu upstream, shift that 1 bit.
Quoted 3 lines
As Lucas said above upstream is different from qemu-kvm unfortunately. On upstream you need to shift return value right for 1 bit to get to the tests return value (why?) and then 0 will be success otherwise failure.
On the why, It was Anthony's call. See, as the upstream test device doesn't have any ports we can write to to make qemu exit, we used another device that can do that. I vaguely remember he argued for the exit code to be shifted by 1 on that other device, and said it wasn't a problem for test suites, since they can define what is a success and what is a failure.