Skip to contentSkip to conversation

kvm-unit-tests: inconsistent test result between run_tests.sh and standalone test

DISCUSSION Guoqing Jiang 2 participants 5 messages Nov 15, 2022 – Nov 16, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I find the two test results (pmu and intel_cet) are quite different, but other test results are consistent.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
...
*PASS pmu (142 tests)*
...
*FAIL intel_cet*
...

1. pmu standalone test
gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu
BUILD_HEAD=73d9d850
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
PASS: core cycles-0
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures
*FAIL pmu (142 tests, 4 unexpected failures)

*And

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat
/usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd /tmp/tmp.jiEHps3KLW enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
*SKIP: No pmu is detected!**
**SUMMARY: 1 tests, 1 skipped*

**
2. intel_cet gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet BUILD_HEAD=73d9d850
*skip intel_cet (test kernel not present)*

Then I am not sure which test result should be correct, standalone test or run all tests. Or am I missed something fundamentally?

Thanks for your reply in advance.

Guoqing

#1 lore raw
Quoted 49 lines · Guoqing Jiang

Hi,

I find the two test results (pmu and intel_cet) are quite different, but other
test results are consistent.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
...
*PASS pmu (142 tests)*
...
*FAIL intel_cet*
...

1. pmu standalone test gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu BUILD_HEAD=73d9d850
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot

(quoted diff) +0 -2
-nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel

/tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
PASS: core cycles-0
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures
*FAIL pmu (142 tests, 4 unexpected failures)

*And

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -2
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd

/tmp/tmp.jiEHps3KLW

Quoted 9 lines

enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
*SKIP: No pmu is detected!**
**SUMMARY: 1 tests, 1 skipped*

./x86-run doesn't look at x86/unittests.cfg, which is where the pmu test states that it needs '-cpu max'. You either need to add it yourself, e.g. './x86-run ./x86/pmu.flat -cpu max' or use run_tests.sh, e.g. './run_tests.sh pmu'. standalone tests get their parameters from x86/unittests.cfg, which is why it's already using '-cpu max'.

Quoted 5 lines

**
2. intel_cet gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet BUILD_HEAD=73d9d850
*skip intel_cet (test kernel not present)*

This error looks like x86/cet.c wasn't built. Maybe do a 'make clean' and 'make standalone' again and watch that cet.c doesn't fail to compile.

Quoted 5 lines

Then I am not sure which test result should be correct, standalone test or run all tests. Or am I missed something fundamentally?

Thanks for your reply in advance.

Thanks,
drew

#2 lore raw

Hi,

Thanks for the quick reply!

Quoted 56 lines · Andrew Jones
Quoted 55 lines · Guoqing Jiang

Hi,

I find the two test results (pmu and intel_cet) are quite different, but other
test results are consistent.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
...
*PASS pmu (142 tests)*
...
*FAIL intel_cet*
...

1. pmu standalone test gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu BUILD_HEAD=73d9d850
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot

(quoted diff) +0 -2
-nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel

/tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
PASS: core cycles-0
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures
*FAIL pmu (142 tests, 4 unexpected failures)

*And

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -2
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd

/tmp/tmp.jiEHps3KLW
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20

Quoted 8 lines
Quoted 3 lines

*SKIP: No pmu is detected!**
**SUMMARY: 1 tests, 1 skipped*

./x86-run doesn't look at x86/unittests.cfg, which is where the pmu test states that it needs '-cpu max'. You either need to add it yourself, e.g. './x86-run ./x86/pmu.flat -cpu max' or use run_tests.sh, e.g. './run_tests.sh pmu'. standalone tests get their parameters from x86/unittests.cfg, which is why it's already using '-cpu max'.

Thanks for the tips. And I still see two different results, one is PASS while another has failures.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests)

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat -cpu max
/usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat -cpu max # -initrd /tmp/tmp.wBVHPW1XUr enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures

Quoted 7 lines

**
2. intel_cet
gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet BUILD_HEAD=73d9d850
*skip intel_cet (test kernel not present)*
This error looks like x86/cet.c wasn't built. Maybe do a 'make clean' and 'make standalone' again and watch that cet.c doesn't fail to compile.

Indeed, seems cet.c is not compiled since only cet.c exists even after run 'make clean' and 'make standalone', but tests/intel_cet is generated.

gjiang@x1:~/source/kvm-unit-tests> ls x86/cet.*
x86/cet.c

Thanks,
Guoqing

#3 lore raw
Quoted 92 lines · Guoqing Jiang

Hi,

Thanks for the quick reply!

Quoted 56 lines · Andrew Jones
Quoted 55 lines · Guoqing Jiang

Hi,

I find the two test results (pmu and intel_cet) are quite different, but other
test results are consistent.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
...
*PASS pmu (142 tests)*
...
*FAIL intel_cet*
...

1. pmu standalone test gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu BUILD_HEAD=73d9d850
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot

(quoted diff) +0 -2
-nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel

/tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
PASS: core cycles-0
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures
*FAIL pmu (142 tests, 4 unexpected failures)

*And

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -2
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd

/tmp/tmp.jiEHps3KLW
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20

Quoted 8 lines
Quoted 3 lines

*SKIP: No pmu is detected!**
**SUMMARY: 1 tests, 1 skipped*

./x86-run doesn't look at x86/unittests.cfg, which is where the pmu test states that it needs '-cpu max'. You either need to add it yourself, e.g. './x86-run ./x86/pmu.flat -cpu max' or use run_tests.sh, e.g. './run_tests.sh pmu'. standalone tests get their parameters from x86/unittests.cfg, which is why it's already using '-cpu max'.

Thanks for the tips. And I still see two different results, one is PASS while another has failures.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests)

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat -cpu max /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -3
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat -cpu max #
-initrd /tmp/tmp.wBVHPW1XUr

enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3

Quoted 8 lines

Fixed counter width: 48
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures

There shouldn't be any difference. Do the exact same failures occur every time with standalone and never with run_tests.sh? I'll bet the failures sometimes occur and sometimes not occur with either test invocation. In which case, the test harness is fine, and the test cases have found something to debug within the test and/or kvm.

Quoted 9 lines
Quoted 8 lines

**
2. intel_cet
gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet BUILD_HEAD=73d9d850
*skip intel_cet (test kernel not present)*
This error looks like x86/cet.c wasn't built. Maybe do a 'make clean' and 'make standalone' again and watch that cet.c doesn't fail to compile.

Quoted 5 lines

Indeed, seems cet.c is not compiled since only cet.c exists even after run 'make clean' and 'make standalone', but tests/intel_cet is generated.

gjiang@x1:~/source/kvm-unit-tests> ls x86/cet.*
x86/cet.c

Take a look at the makefile. It shows that your compiler needs to support -fcf-protection=full.

We could consider a patch for mkstandalone like below, but I think I prefer the way it is now, rather than silently dropping tests.

Thanks,
drew

scripts/mkstandalone.sh +5 -4
index 86c7e5498246..3d61f24d3d6e 100755
@@ -47,9 +47,7 @@ generate_test ()
        echo "echo BUILD_HEAD=$(cat build-head)"
 
        if [ ! -f $kernel ]; then
-               echo 'echo "skip '"$testname"' (test kernel not present)"'
-               echo 'exit 2'
-               return
+               return 2
        fi
 
        echo "trap 'rm -f \$cleanup' EXIT"
@@ -89,7 +87,10 @@ function mkstandalone()
 
        standalone=tests/$testname
 
-       generate_test "$@" > $standalone
+       if ! generate_test "$@" > $standalone; then
+               rm -f $standalone
+               return
+       fi
 
        chmod +x $standalone
        echo Written $standalone.
#4 lore raw
Quoted 97 lines · Andrew Jones
Quoted 96 lines · Guoqing Jiang

Hi,

Thanks for the quick reply!

Quoted 56 lines · Andrew Jones
Quoted 55 lines · Guoqing Jiang

Hi,

I find the two test results (pmu and intel_cet) are quite different, but other
test results are consistent.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
...
*PASS pmu (142 tests)*
...
*FAIL intel_cet*
...

1. pmu standalone test gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu BUILD_HEAD=73d9d850
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot

(quoted diff) +0 -2
-nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel

/tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
PASS: core cycles-0
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2
FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures
*FAIL pmu (142 tests, 4 unexpected failures)

*And

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -2
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd

/tmp/tmp.jiEHps3KLW
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20

Quoted 8 lines
Quoted 3 lines

*SKIP: No pmu is detected!**
**SUMMARY: 1 tests, 1 skipped*

./x86-run doesn't look at x86/unittests.cfg, which is where the pmu test states that it needs '-cpu max'. You either need to add it yourself, e.g. './x86-run ./x86/pmu.flat -cpu max' or use run_tests.sh, e.g. './run_tests.sh pmu'. standalone tests get their parameters from x86/unittests.cfg, which is why it's already using '-cpu max'.

Thanks for the tips. And I still see two different results, one is PASS while another has failures.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests)

gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat -cpu max /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev

(quoted diff) +0 -3
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
-device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat -cpu max #
-initrd /tmp/tmp.wBVHPW1XUr

enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
PMU version:         2
GP counters:         4
GP counter width:    48
Mask length:         7
Fixed counters:      3
Fixed counter width: 48
...
FAIL: llc misses-0
FAIL: llc misses-1
FAIL: llc misses-2

Quoted 8 lines
Quoted 3 lines

FAIL: llc misses-3
...
SUMMARY: 142 tests, 4 unexpected failures

There shouldn't be any difference. Do the exact same failures occur every time with standalone and never with run_tests.sh? I'll bet the failures sometimes occur and sometimes not occur with either test invocation. In which case, the test harness is fine, and the test cases have found something to debug within the test and/or kvm.

The pmu standalone test always has failures though the failure numbers are not same after every run. And './run_tests.sh pmu' which rarely reports failure.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu FAIL pmu (142 tests, 4 unexpected failures) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests) gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu PASS pmu (142 tests)

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

**
2. intel_cet
gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet BUILD_HEAD=73d9d850
*skip intel_cet (test kernel not present)*
This error looks like x86/cet.c wasn't built. Maybe do a 'make clean' and 'make standalone' again and watch that cet.c doesn't fail to compile.

Indeed, seems cet.c is not compiled since only cet.c exists even after run 'make clean' and 'make standalone', but tests/intel_cet is generated.

Quoted 7 lines
Quoted 2 lines

gjiang@x1:~/source/kvm-unit-tests> ls x86/cet.*
x86/cet.c

Take a look at the makefile. It shows that your compiler needs to support -fcf-protection=full.

We could consider a patch for mkstandalone like below, but I think I prefer the way it is now, rather than silently dropping tests.

Thanks for the patch, is it possible to print more intuitive information instead of 'test kernel not present'? I think access-reduced-maxphyaddr is better.

gjiang@x1:~/source/kvm-unit-tests> ./tests/access-reduced-maxphyaddr
BUILD_HEAD=73d9d850
SKIP access-reduced-maxphyaddr (/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr not equal to Y)

Quoted 20 lines

Thanks,
drew

scripts/mkstandalone.sh +1 -3
index 86c7e5498246..3d61f24d3d6e 100755
@@ -47,9 +47,7 @@ generate_test ()
         echo "echo BUILD_HEAD=$(cat build-head)"
 
         if [ ! -f $kernel ]; then
-               echo 'echo "skip '"$testname"' (test kernel not present)"'
-               echo 'exit 2'
-               return
+               return 2
         fi
 
         echo "trap 'rm -f \$cleanup' EXIT"
@@ -89,7 +87,10 @@ function mkstandalone()
 

scripts/mkstandalone.sh function mkstandalone() · line 96

Quoted 10 lines
scripts/mkstandalone.sh +4 -1
         standalone=tests/$testname
 
-       generate_test "$@" > $standalone
+       if ! generate_test "$@" > $standalone; then
+               rm -f $standalone
+               return
+       fi
 
         chmod +x $standalone
         echo Written $standalone.

Thanks,
Guoqing