kvm-unit-tests failures

Hi,

We are planning on running kvm-unit-tests as part of our test suite; but I've noticed that many tests fail (even running the latest kvm tip). After searching I found many BZ entires that seem to point at this master bug for tracking these issues:
https://bugzilla.redhat.com/show_bug.cgi?id=1079979
However, this bug is private; and cannot be viewed by the public.

I'd like to know how to help report issues that we observe with testing in order to help fix these tests, or understand any progress being made to fix them already. Is there a public bug that everybody can view to track these issues? Should I be reporting new bugs with failures in the unit tests? Where is the appropriate place to file bugs against kvm-unit-tests and discuss issues?

Thanks,
--chris j arges

Il 01/08/2014 23:09, Chris J Arges ha scritto:

This bug is for failures in the RHEL7 kernel, which is older than kvm.git. Which tests fail for you?.

kvm.git bugs are tracked in bugzilla.kernel.org, but for now I think it's better if you just reply here.

Paolo

<snip>

Paulo,
Hopefully this is enough to get started; I can file this as a bug if its easier to track. Let me know if there is additional information that is needed. I'm thinking some may be userspace qemu-system-x86_64 issues since three of them can be resolved by adding "-cpu host"; however it may be something to add in the test-suite. Let me know and I am happy to write a patch depending on what you think will work.

All tests below are done with the kvm tip (fresh as of today) merged with 3.16-rc4.

Thanks,
--chris j arges

--

* Failures:

- pmu

$ sudo ./x86-run x86/pmu.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/pmu.flat -smp 1
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
PMU version:         33
GP counters:         1
GP counter width:    0
Mask length:         0
Fixed counters:      1
Fixed counter width: 0
GP event 'core cycles' is disabled
GP event 'instructions' is disabled
GP event 'ref cycles' is disabled
GP event 'llc refference' is disabled
GP event 'llc misses' is disabled
GP event 'branches' is disabled
unhandled excecption 13
Return value from qemu: 15

As suggested in comment 2 on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1079796
Adding the -cpu host allows this test to PASS.

- tsc_adjust

$ sudo ./x86-run x86/tsc_adjust.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/tsc_adjust.flat -smp 1
enabling apic
failure: IA32_TSC_ADJUST msr read / write incorrect
failure: TSC did not adjust for IA32_TSC_ADJUST value
failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
Return value from qemu: 3

Here again adding "-cpu host" allows this to pass.

- pcid

$ sudo ./x86-run x86/tsc_adjust.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/tsc_adjust.flat -smp 1
enabling apic
failure: IA32_TSC_ADJUST msr read / write incorrect
failure: TSC did not adjust for IA32_TSC_ADJUST value
failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
Return value from qemu: 3
ubuntu@zoot:~/kvm-unit-tests$ sudo ./x86-run x86/pcid.flat -smp 1 -cpu
qemu64,+pcid
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/pcid.flat -smp 1 -cpu qemu64,+pcid
enabling apic
PASS: CPUID consistency
FAIL: Test on PCID when enabled
PASS: Test on INVPCID when disabled

SUMMARY: 3 tests, 1 unexpected failures
Return value from qemu: 3

Here again adding "-cpu host" allows this to pass.

- msr (HANG)
./x86-run x86/msr.flat -smp 1

$ sudo ./x86-run x86/msr.flat -smp 1qemu-system-x86_64 -enable-kvm
-device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-display none -serial stdio -device pci-testdev -kernel x86/msr.flat -smp 1
enabling apic
enabling apic
enabling apic

"enabling apic" repeats and adding -cpu host seems to have no effect. Interestingly enough on an Ubuntu 3.13 series kernel this PASSes.

- vmx (HANG)

$ sudo ./x86-run x86/vmx.flat -smp 1 -cpu host,+vmx
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/vmx.flat -smp 1 -cpu host,+vmx
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
PASS: test vmxon with FEATURE_CONTROL cleared
PASS: test vmxon without FEATURE_CONTROL lock
PASS: test enable VMX in FEATURE_CONTROL
PASS: test FEATURE_CONTROL lock bit
PASS: test vmxon with unaligned vmxon region
PASS: test vmxon with bits set beyond physical address width
PASS: test vmxon with invalid revision identifier
PASS: test vmxon with valid vmxon region
PASS: test vmptrld with unaligned vmcs
PASS: test vmptrld with vmcs address bits set beyond physical address width
PASS: test vmptrld with vmxon region
PASS: test vmptrld with valid vmcs region
PASS: test vmclear with unaligned vmcs
PASS: test vmclear with vmcs address bits set beyond physical address width
PASS: test vmclear with vmxon region
PASS: test vmclear with valid vmcs region
PASS: test vmptrst
PASS: test vmxoff

Test suite: VMX capability reporting
PASS: MSR_IA32_VMX_BASIC
PASS: MSR_IA32_VMX_MISC
PASS: MSR_IA32_VMX_PINBASED_CTLS
PASS: MSR_IA32_VMX_PROCBASED_CTLS
PASS: MSR_IA32_VMX_PROCBASED_CTLS2
PASS: MSR_IA32_VMX_EXIT_CTLS
PASS: MSR_IA32_VMX_ENTRY_CTLS
PASS: MSR_IA32_VMX_IA32_VMX_CR0_FIXED0/1
PASS: MSR_IA32_VMX_IA32_VMX_CR4_FIXED0/1
PASS: MSR_IA32_VMX_VMCS_ENUM
PASS: MSR_IA32_VMX_EPT_VPID_CAP

Test suite: vmenter
PASS: test vmlaunch
PASS: test vmresume

Test suite: preemption timer
PASS: Keep preemption value
PASS: Save preemption value
PASS: busy-wait for preemption timer
PASS: preemption timer during hlt
PASS: preemption timer with 0 value

Test suite: control field PAT
PASS: Exit save PAT
PASS: Exit load PAT
PASS: Entry load PAT

Test suite: control field EFER
PASS: Exit save EFER
PASS: Exit load EFER
PASS: Entry load EFER

Test suite: CR shadowing
PASS: Read through CR0
PASS: Read through CR4
PASS: Write through CR0
PASS: Write through CR4
PASS: Read shadowing CR0
PASS: Read shadowing CR4
PASS: Write shadowing CR0 (same value)
PASS: Write shadowing CR4 (same value)
PASS: Write shadowing different X86_CR0_TS
PASS: Write shadowing different X86_CR0_MP
PASS: Write shadowing different X86_CR4_TSD
PASS: Write shadowing different X86_CR4_DE

Test suite: I/O bitmap
PASS: I/O bitmap - I/O pass
PASS: I/O bitmap - I/O width, byte
PASS: I/O bitmap - I/O direction, in
PASS: I/O bitmap - trap in
PASS: I/O bitmap - I/O width, word
PASS: I/O bitmap - I/O direction, out
PASS: I/O bitmap - trap out
PASS: I/O bitmap - I/O width, long
PASS: I/O bitmap - I/O port, low part
PASS: I/O bitmap - I/O port, high part
PASS: I/O bitmap - partial pass
PASS: I/O bitmap - overrun
PASS: I/O bitmap - ignore unconditional exiting
PASS: I/O bitmap - unconditional exiting

Test suite: instruction intercept
PASS: execute HLT
PASS: intercept HLT
PASS: execute INVLPG
PASS: intercept INVLPG
PASS: execute MWAIT
PASS: intercept MWAIT
PASS: execute RDPMC
PASS: intercept RDPMC
PASS: execute RDTSC
PASS: intercept RDTSC
PASS: execute CR3 load
PASS: intercept CR3 load
PASS: execute CR3 store
PASS: intercept CR3 store
PASS: execute CR8 load
PASS: intercept CR8 load
PASS: execute CR8 store
PASS: intercept CR8 store
PASS: execute MONITOR
PASS: intercept MONITOR
PASS: execute PAUSE
PASS: intercept PAUSE
PASS: execute WBINVD
PASS: intercept WBINVD
PASS: intercept CPUID
PASS: intercept INVD

Test suite: EPT framework
PASS: EPT basic framework
PASS: EPT misconfigurations
PASS: EPT violation - page permission
FAIL: EPT violation - paging structure

Test suite: interrupt
PASS: direct interrupt while running guest
PASS: intercepted interrupt while running guest
PASS: direct interrupt + hlt

Here it just hangs.

* KVM Related Config:

CONFIG_KVM_GUEST=y
CONFIG_KVM_DEBUG_FS=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_KVM_DEVICE_ASSIGNMENT=y

* Module parameters:

$ tail /sys/module/kvm/parameters/*
==> /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts <==
N

==> /sys/module/kvm/parameters/ignore_msrs <==
N

==> /sys/module/kvm/parameters/min_timer_period_us <== 500

==> /sys/module/kvm/parameters/tsc_tolerance_ppm <== 250

$ tail /sys/module/kvm_intel/parameters/*
==> /sys/module/kvm_intel/parameters/emulate_invalid_guest_state <==
Y

==> /sys/module/kvm_intel/parameters/enable_apicv <== Y

==> /sys/module/kvm_intel/parameters/enable_shadow_vmcs <== Y

==> /sys/module/kvm_intel/parameters/ept <==
Y

==> /sys/module/kvm_intel/parameters/eptad <==
Y

==> /sys/module/kvm_intel/parameters/fasteoi <==
Y

==> /sys/module/kvm_intel/parameters/flexpriority <== Y

==> /sys/module/kvm_intel/parameters/nested <==
Y

==> /sys/module/kvm_intel/parameters/ple_gap <==
128

==> /sys/module/kvm_intel/parameters/ple_window <== 4096

==> /sys/module/kvm_intel/parameters/unrestricted_guest <== Y

==> /sys/module/kvm_intel/parameters/vmm_exclusive <== Y

==> /sys/module/kvm_intel/parameters/vpid <==
Y

Il 04/08/2014 19:02, Chris J Arges ha scritto:

Are you aware of x86/unittests.cfg and run_tests.sh?

Paolo

Paolo,
Yes this is how I initially ran the script:
./configure && make
sudo ./run_tests.sh -v

Then if any commands that failed, I re-ran the command it output in order to show more output here for debugging.

--chris j arges

Il 04/08/2014 20:37, Chris J Arges ha scritto:

Which tests are failing and which require "-cpu host"?

Paolo

Paolo,
I included these results with my original reply:
https://www.mail-archive.com/[email protected]/msg105950.html
See the "Failures" part.

Thanks,
--chris j arges

Il 04/08/2014 19:02, Chris J Arges ha scritto:

Sorry, I had missed the list of failures.

Correct, I thought unittests.cfg already did that.

Looks like QEMU doesn't know the tsc_adjust flag at all. CCing Eduardo.

It passes here, and fails with "-cpu host":

PASS: CPUID consistency
FAIL: Test on PCID when enabled
PASS: Test on INVPCID when disabled

Looks like the "Test on PCID when enabled" has problems, and the -cpu flag is a red herring.

The APICBASE register is already tested by the apic testcase, and the simple-minded tests in msr.flat actually trigger a general protection fault nowadays because the quality of the emulation improved. I'm dropping the APICBASE tests from msr.flat.

It fails here for me. It could be related to APICv, can you try with the enable_apic=0 parameter to kvm-intel.ko?

Paolo

QEMU doesn't know the flag name yet (although we have migration support for the MSR since 2012).

Note that with the new "migratable=yes" default on QEMU 2.1, an unknown feature won't be enabled even when using "-cpu host", and will require "-cpu host,migratable=no".

I see a bug on the unit test code: it is calling cpuid(7) without checking if (cpuid(0).a >= 7) first, so it is checking meaningless CPUID bits when cpuid_level < 7. This causes the following problem:

[kvm-unit-tests/master=]$ ./x86-run x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=7
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=7 enabling apic
enabling apic
success: IA32_TSC_ADJUST feature not enabled
Return value from qemu: 1
[kvm-unit-tests/master=]$ ./x86-run x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=4
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=4 enabling apic
enabling apic
failure: IA32_TSC_ADJUST msr read / write incorrect
failure: TSC did not adjust for IA32_TSC_ADJUST value
failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
Return value from qemu: 3

Eduardo

Updated information with the three new kvm-unit-tests patches you've posted.

Ok with your latest patches, the test case runs, but fails on a few other tests:
⟫ sudo ./x86-run x86/pmu.flat -smp 1 -cpu host | grep FAIL
FAIL: fixed-2
FAIL: all counters

This test now passes.

This test now passes.

With enable_apicv=0, this test case just FAILs for me.

⟫ sudo ./x86-run x86/vmx.flat -smp 1 -cpu host,+vmx | grep FAIL
FAIL: EPT violation - paging structure
FAIL: intercepted interrupt + hlt

I'll try to dig a bit more, but things are looking much better! Thanks for the help and patches.
--chris j arges