New: kvm-unit-tests vmx has about 60% of failure chance
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Bug ID: 216177
Summary: kvm-unit-tests vmx has about 60% of failure chance
Product: Virtualization
Version: unspecified
Kernel Version: 5.19-rc1
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoCreated attachment 301281
--> https://bugzilla.kernel.org/attachment.cgi?id=301281&action=edit
vmx failure log
Environment:
CPU Architecture: x86_64
Host OS: Red Hat Enterprise Linux 8.4 (Ootpa)
Host kernel: 5.19.0-rc1
gcc: gcc version 8.4.1
Host kernel source: https://git.kernel.org/pub/scm/virt/kvm/kvm.git
Branch: next
Commit: 4b88b1a518b337de1252b8180519ca4c00015c9e
Qemu source: https://git.qemu.org/git/qemu.git
Branch: master
Commit: 40d522490714b65e0856444277db6c14c5cc3796
kvm-unit-tests source: https://gitlab.com/kvm-unit-tests/kvm-unit-tests.git
Branch: master
Commit: ca85dda2671e88d34acfbca6de48a9ab32b1810d
Bug Detailed Description:
kvm-unit-tests vmx has about 60% of chance to fail. In my case, failure happened 6 times out of 10 times of tests.
Reproducing Steps:
rmmod kvm_intel
modprobe kvm_intel nested=Y
git clone https://gitlab.com/kvm-unit-tests/kvm-unit-tests.git
cd kvm-unit-tests
./configure
make standalone
cd tests
./vmx -cpu host
Actual Result:
...
SUMMARY: 430101 tests, 1 unexpected failures, 2 expected failures, 4 skipped FAIL vmx (430101 tests, 1 unexpected failures, 2 expected failures, 4 skipped)
Expected Result:
...
SUMMARY: 430101 tests, 2 expected failures, 4 skipped PASS vmx (430101 tests, 2 expected failures, 4 skipped)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC it's KVM that's at fault).
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
Quoted 12 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
Quoted 6 lines
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
For the record:
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #2 from Nadav Amit ([email protected]) ---
Quoted 10 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
Quoted 7 lines
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
For the record:
https://lore.kernel.org/kvm/[email protected]/
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 16 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 7 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).
Quoted 6 lines
Quoted 2 lines
Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)For the record:
Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
Quoted 27 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 19 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 9 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Quoted 6 lines
Quoted 3 lines
For the record:
Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?
As I noted, at some point I did not manage to reproduce the failure.
The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.
Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
Quoted 3 lines · Nadav Amit
The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.
KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.
Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #4 from Sean Christopherson ([email protected]) ---
Quoted 24 lines · On Tue, Jun 28, 2022, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 10 lines
Quoted 9 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Quoted 6 lines
Quoted 3 lines
For the record:
Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?
No, AFAIK no one has posted a fix. If it's the KVM issue I'm thinking of, the fix is non-trivial. It'd require scheduling a timer in L0 with a deadline shorter
than what L1 requests when emulating the VMX timer, and then busy waiting in L0 if
the host timer fires early. KVM already does this for e.g. L1's TSC deadline timer.
That code would need to be adapated for the nested VMX preemption timer.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #5 from Nadav Amit ([email protected]) ---
Quoted 25 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 19 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 9 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Quoted 6 lines
Quoted 3 lines
For the record:
Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?
As I noted, at some point I did not manage to reproduce the failure.
The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #6 from Sean Christopherson ([email protected]) ---
Quoted 36 lines · On Tue, Jun 28, 2022, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #5 from Nadav Amit ([email protected]) ---
Quoted 15 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)For the record:
https://lore.kernel.org/kvm/[email protected]/
Quoted 2 lines
Thanks for your reply. So this is a KVM bug, and you have sent a patch to
kvm
Quoted 8 lines
Quoted 1 line
to fix this bug, right?
As I noted, at some point I did not manage to reproduce the failure.
The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be
a KVM bug.
Oooh, your failure was on bare-metal. I didn't grok that. Though it could be both a hardware bug and a KVM bug :-)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #7 from Yang Lixiao ([email protected]) --- (In reply to Sean Christopherson from comment #6)
Quoted 45 lines · On Tue, Jun 28, 2022, [email protected] wrote:
Quoted 19 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #5 from Nadav Amit ([email protected]) ---
Quoted 15 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)
Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CC| |[email protected]
--- Comment #1 from Sean Christopherson ([email protected]) --- It's
vmx_preemption_timer_expiry_test, which is known to be flaky(though
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
IIRC
it's KVM that's at fault).Test suite:
vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)For the record:
https://lore.kernel.org/kvm/[email protected]/
Quoted 1 line
Quoted 7 lines
Quoted 1 line
Thanks for your reply. So this is a KVM bug, and you have sent a patch to
kvm
Quoted 1 line
to fix this bug, right?
As I noted, at some point I did not manage to reproduce the failure.
The failure on bare-metal that I experienced hints that this is either a
test
Quoted 8 lines
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is likely
to
Quoted 2 lines
be
a KVM bug.Oooh, your failure was on bare-metal. I didn't grok that. Though it could be
both a hardware bug and a KVM bug :-)
In my tests, I tested kvm-unit-tests vmx on bare-metal (not on VM) and this bug happened on two different Ice Lake machines and one Cooper Lake machine.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #8 from Jim Mattson ([email protected]) ---
Quoted 4 lines · Nadav Amit
The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be
a KVM bug.
KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.
Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 31 lines · Nadav Amit
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is either a
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is likely
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)
On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
Quoted 45 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 31 lines · Nadav Amit
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is either a
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is likely
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Quoted 7 lines
On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)
Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #10 from Jim Mattson ([email protected]) ---
Quoted 45 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 31 lines · Nadav Amit
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is either a
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is likely
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)
Quoted 7 lines
On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)
Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)
Quoted 52 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 8 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 3 lines · Nadav Amit
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is either
a
Quoted 2 lines
Quoted 2 lines
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is
likely
Quoted 39 lines
Quoted 26 lines
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)
Quoted 5 lines
Quoted 3 lines
On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)Wow! Those are *huge* overruns. What is the value of MSR
0x9Bon these hosts?
All of the values of MSR 0x9B on the three hosts are 0.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
Quoted 63 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)
Quoted 52 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 8 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 3 lines · Nadav Amit
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is either
a
Quoted 2 lines
Quoted 2 lines
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is
likely
Quoted 39 lines
Quoted 26 lines
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)Quoted 5 lines
Quoted 3 lines
On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)Wow! Those are *huge* overruns. What is the value of MSR
0x9Bon these hosts?
Quoted 7 lines
All of the values of MSR
0x9Bon the three hosts are 0.--
You may reply to this email to add a comment.You are receiving this mail because:
You are watching the assignee of the bug.
Doh! There is a glaring bug in the test. I'll post a fix soon.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #12 from Jim Mattson ([email protected]) ---
Quoted 66 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)
Quoted 7 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 1 line
On Mon, Jun 27, 2022 at 8:54 PM Nadav Amit <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is
either
Quoted 43 lines
a
Quoted 2 lines
Quoted 2 lines
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is
likely
Quoted 39 lines
Quoted 26 lines
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Quoted 4 lines
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)Quoted 5 lines
Quoted 3 lines
On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)Wow! Those are *huge* overruns. What is the value of MSR
0x9Bon thesehosts?
Quoted 7 lines
All of the values of MSR
0x9Bon the three hosts are 0.--
You may reply to this email to add a comment.You are receiving this mail because:
You are watching the assignee of the bug.
Doh! There is a glaring bug in the test. I'll post a fix soon.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #13 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #12)
Quoted 68 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:
Quoted 9 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216177
--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)
Quoted 4 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 3 lines
Quoted 32 lines
Quoted 3 lines
Quoted 3 lines
--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)
Quoted 1 line
On Mon, Jun 27, 2022 at 8:54 PM Nadav Amit <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The failure on bare-metal that I experienced hints that this is
either
Quoted 25 lines
a
Quoted 2 lines
Quoted 2 lines
test
Quoted 1 line
bug or (much less likely) a hardware bug. But I do not think it is
likely
Quoted 21 lines
Quoted 21 lines
to
Quoted 2 lines
be
a KVM bug.KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit
f4124500c2c1("KVM: nVMX: Fully emulate preemption timer"), which uses an L0CLOCK_MONOTONIChrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that theCLOCK_MONOTONICtimer is subject to time slew.Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.
I can't explain the bare-metal failures, but I will note that the
test
Quoted 25 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.
Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?
On the first Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)On the second Ice lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)Quoted 5 lines
Quoted 3 lines
On Cooper lake:
Test suite:vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)Wow! Those are *huge* overruns. What is the value of MSR
0x9Bon thesehosts?
Quoted 8 lines
Quoted 7 lines
All of the values of MSR
0x9Bon the three hosts are 0.--
You may reply to this email to add a comment.You are receiving this mail because:
You are watching the assignee of the bug.Doh! There is a glaring bug in the test. I'll post a fix soon.
Thanks!
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.