The "memory" test is failing in the kvm-unit-tests CI
Hi,
I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.
It used to work fine with commit 2480430a here in January:
https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873
Now I've re-run the CI with the same commit 2480430a here and it is failing now:
https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733
Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?
Thomas
Sean Christopherson
reviewerQuoted 7 lines · Thomas Huth
Hi,
I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.
Quoted 7 lines
It used to work fine with commit
2480430ahere in January:https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873
Now I've re-run the CI with the same commit
2480430ahere and it is failing now:https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733
Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.
Quoted 3 lines
Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?
Nothing jumps to mind. Triaging this without at least the logs in going to be painful.
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth
Hi,
I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.
It used to work fine with commit
2480430ahere in January:
Quoted 8 lines
Quoted 5 lines
https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873
Now I've re-run the CI with the same commit
2480430ahere and it is failing now:https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733
Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.
Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.
Meanwhile, here's a run with a manual "cat logs/memory.log":
https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726
Seems like these are the failing memory tests:
FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)
Thomas
Quoted 7 lines
Quoted 3 lines
Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?
Nothing jumps to mind. Triaging this without at least the logs in going to be painful.
Sean Christopherson
reviewerQuoted 23 lines · Thomas Huth
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth
Hi,
I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.
It used to work fine with commit
2480430ahere in January:Quoted 8 lines
Quoted 5 lines
https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873
Now I've re-run the CI with the same commit
2480430ahere and it is failing now:https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733
Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.
Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.
Quoted 8 lines
Meanwhile, here's a run with a manual "cat logs/memory.log":
https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726
Seems like these are the failing memory tests:
FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)
More than likely what is happening is that the platform supports CLFLUSHOPT and CLWB (possibly even via a ucode patch update), but the CPUID bits are not being enumerated to the guest. Neither VMX nor SVM has intercept controls for the instructions, so KVM has no way to enforce the the guest's CPUID model. E.g. the failures can be reproduce by manually hiding the features:
rkt ./x86/run x86/memory.flat -smp 1 -cpu max,-clflushopt,-clwb
This isn't a KVM bug because of the virtualization hole. And really, the test itself is bogus when running on KVM precisely because of said hole (similar holes exist for all the other instructions in the test).
The test appears to have been added for QEMU's TCG, which makes sense as there shouldn't be any virtualization holes in a pure emulation environment.
That said, it is interesting that the test is suddenly failing, as it means something is buggy. If you can run commands on the host, check for host support via /proc/cpuinfo. If those come back negative (no support), then it would appear that hardware or the host kernel is in a bad/unexpected state.
grep -q clflushopt /proc/cpuinfo
grep -q clwb /proc/cpuinfo
Quoted 48 lines · Sean Christopherson
Quoted 23 lines · Thomas Huth
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth
Hi,I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.
It used to work fine with commit
2480430ahere in January:Quoted 8 lines
Quoted 5 lines
https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873Now I've re-run the CI with the same commit
2480430ahere and it is failing now:https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.
Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.
Quoted 8 lines
Meanwhile, here's a run with a manual "cat logs/memory.log":
https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726
Seems like these are the failing memory tests:
FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)More than likely what is happening is that the platform supports CLFLUSHOPT and CLWB (possibly even via a ucode patch update), but the CPUID bits are not being enumerated to the guest. Neither VMX nor SVM has intercept controls for the instructions, so KVM has no way to enforce the the guest's CPUID model. E.g. the failures can be reproduce by manually hiding the features:
rkt ./x86/run x86/memory.flat -smp 1 -cpu max,-clflushopt,-clwbThis isn't a KVM bug because of the virtualization hole. And really, the test itself is bogus when running on KVM precisely because of said hole (similar holes exist for all the other instructions in the test).
The test appears to have been added for QEMU's TCG, which makes sense as there shouldn't be any virtualization holes in a pure emulation environment.
Quoted 7 lines
That said, it is interesting that the test is suddenly failing, as it means something is buggy. If you can run commands on the host, check for host support via
/proc/cpuinfo. If those come back negative (no support), then it would appear that hardware or the host kernel is in a bad/unexpected state.grep -q clflushopt
/proc/cpuinfo
grep -q clwb/proc/cpuinfo
I dumped the cpuinfo here:
https://cirrus-ci.com/task/4861043097206784?logs=main#L22
And indeed, clflushopt and clwb do not show up. It's a nested setup, so I guess the flags have been disabled on the L0 host already.
I guess there's not much we can do here except disabling the "memory" test on cirrus-CI now...
Thomas