Skip to contentSkip to conversation

The "memory" test is failing in the kvm-unit-tests CI

DISCUSSION Thomas Huth 2 participants 5 messages Mar 29, 2023 – Apr 3, 2023 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.

It used to work fine with commit 2480430a here in January:

https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873

Now I've re-run the CI with the same commit 2480430a here and it is failing now:

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733

Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?

Thomas

#1 lore raw
Quoted 7 lines · Thomas Huth

Hi,

I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.

Quoted 7 lines

It used to work fine with commit 2480430a here in January:

https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873

Now I've re-run the CI with the same commit 2480430a here and it is failing now:

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733

Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.

Quoted 3 lines

Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?

Nothing jumps to mind. Triaging this without at least the logs in going to be painful.

#2 lore raw
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth

Hi,

I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.

It used to work fine with commit 2480430a here in January:

Quoted 8 lines
Quoted 5 lines

https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873

Now I've re-run the CI with the same commit 2480430a here and it is failing now:

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733

Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.

Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.

Meanwhile, here's a run with a manual "cat logs/memory.log":

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726

Seems like these are the failing memory tests:

FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)

Thomas

Quoted 7 lines
Quoted 3 lines

Does anybody have an idea what could be causing this regression? The build in January used 7.0.0-12.fc37, the new build used 7.0.0-15.fc37, could that be related? Or maybe a different kernel version?

Nothing jumps to mind. Triaging this without at least the logs in going to be painful.

#3 lore raw
Quoted 23 lines · Thomas Huth
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth

Hi,

I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.

It used to work fine with commit 2480430a here in January:

Quoted 8 lines
Quoted 5 lines

https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873

Now I've re-run the CI with the same commit 2480430a here and it is failing now:

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733

Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.

Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.

Quoted 8 lines

Meanwhile, here's a run with a manual "cat logs/memory.log":

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726

Seems like these are the failing memory tests:

FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)

More than likely what is happening is that the platform supports CLFLUSHOPT and CLWB (possibly even via a ucode patch update), but the CPUID bits are not being enumerated to the guest. Neither VMX nor SVM has intercept controls for the instructions, so KVM has no way to enforce the the guest's CPUID model. E.g. the failures can be reproduce by manually hiding the features:

rkt ./x86/run x86/memory.flat -smp 1 -cpu max,-clflushopt,-clwb

This isn't a KVM bug because of the virtualization hole. And really, the test itself is bogus when running on KVM precisely because of said hole (similar holes exist for all the other instructions in the test).

The test appears to have been added for QEMU's TCG, which makes sense as there shouldn't be any virtualization holes in a pure emulation environment.

That said, it is interesting that the test is suddenly failing, as it means something is buggy. If you can run commands on the host, check for host support via /proc/cpuinfo. If those come back negative (no support), then it would appear that hardware or the host kernel is in a bad/unexpected state.

grep -q clflushopt /proc/cpuinfo
grep -q clwb /proc/cpuinfo

#4 lore raw
Quoted 48 lines · Sean Christopherson
Quoted 23 lines · Thomas Huth
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Thomas Huth
   Hi,

I noticed that in recent builds, the "memory" test started failing in the kvm-unit-test CI. After doing some experiments, I think it might rather be related to the environment than to a recent change in the k-u-t sources.

It used to work fine with commit 2480430a here in January:

Quoted 8 lines
Quoted 5 lines
   https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/jobs/3613156199#L2873

Now I've re-run the CI with the same commit 2480430a here and it is failing now:

   https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4022074711#L2733

Can you provide the logs from the failing test, and/or the build artifacts? I tried, and failed, to find them on Gitlab.

Yes, that's still missing in the CI scripts ... I'll try to come up with a patch that provides the logs as artifacts.

Quoted 8 lines

Meanwhile, here's a run with a manual "cat logs/memory.log":

https://gitlab.com/thuth/kvm-unit-tests/-/jobs/4029213352#L2726

Seems like these are the failing memory tests:

FAIL: clflushopt (ABSENT)
FAIL: clwb (ABSENT)

More than likely what is happening is that the platform supports CLFLUSHOPT and CLWB (possibly even via a ucode patch update), but the CPUID bits are not being enumerated to the guest. Neither VMX nor SVM has intercept controls for the instructions, so KVM has no way to enforce the the guest's CPUID model. E.g. the failures can be reproduce by manually hiding the features:

   rkt ./x86/run x86/memory.flat -smp 1 -cpu max,-clflushopt,-clwb

This isn't a KVM bug because of the virtualization hole. And really, the test itself is bogus when running on KVM precisely because of said hole (similar holes exist for all the other instructions in the test).

The test appears to have been added for QEMU's TCG, which makes sense as there shouldn't be any virtualization holes in a pure emulation environment.

Quoted 7 lines

That said, it is interesting that the test is suddenly failing, as it means something is buggy. If you can run commands on the host, check for host support via /proc/cpuinfo. If those come back negative (no support), then it would appear that hardware or the host kernel is in a bad/unexpected state.

grep -q clflushopt /proc/cpuinfo
grep -q clwb /proc/cpuinfo

I dumped the cpuinfo here:

https://cirrus-ci.com/task/4861043097206784?logs=main#L22

And indeed, clflushopt and clwb do not show up. It's a nested setup, so I guess the flags have been disabled on the L0 host already.

I guess there's not much we can do here except disabling the "memory" test on cirrus-CI now...

Thomas