Skip to contentSkip to conversation

New: kvm-unit-tests vmx has about 60% of failure chance

Bug 216177 [email protected] 3 participants 19 messages Jun 27, 2022 – Jun 29, 2022 KVM

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=216177

            Bug ID: 216177
           Summary: kvm-unit-tests vmx has about 60% of failure chance
           Product: Virtualization
           Version: unspecified
    Kernel Version: 5.19-rc1
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 301281
--> https://bugzilla.kernel.org/attachment.cgi?id=301281&action=edit
vmx failure log

Environment:
CPU Architecture: x86_64
Host OS: Red Hat Enterprise Linux 8.4 (Ootpa)
Host kernel: 5.19.0-rc1
gcc: gcc version 8.4.1
Host kernel source: https://git.kernel.org/pub/scm/virt/kvm/kvm.git
Branch: next
Commit: 4b88b1a518b337de1252b8180519ca4c00015c9e

Qemu source: https://git.qemu.org/git/qemu.git
Branch: master
Commit: 40d522490714b65e0856444277db6c14c5cc3796

kvm-unit-tests source: https://gitlab.com/kvm-unit-tests/kvm-unit-tests.git
Branch: master
Commit: ca85dda2671e88d34acfbca6de48a9ab32b1810d

Bug Detailed Description:
kvm-unit-tests vmx has about 60% of chance to fail. In my case, failure happened 6 times out of 10 times of tests.

Reproducing Steps:
rmmod kvm_intel
modprobe kvm_intel nested=Y
git clone https://gitlab.com/kvm-unit-tests/kvm-unit-tests.git
cd kvm-unit-tests
./configure
make standalone
cd tests
./vmx -cpu host

Actual Result:
...
SUMMARY: 430101 tests, 1 unexpected failures, 2 expected failures, 4 skipped FAIL vmx (430101 tests, 1 unexpected failures, 2 expected failures, 4 skipped)

Expected Result:
...
SUMMARY: 430101 tests, 2 expected failures, 4 skipped PASS vmx (430101 tests, 2 expected failures, 4 skipped)

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

Quoted 12 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

Quoted 6 lines

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

For the record:

https://lore.kernel.org/kvm/[email protected]/

#3 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #2 from Nadav Amit ([email protected]) ---

Quoted 10 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

Quoted 7 lines

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

For the record:

https://lore.kernel.org/kvm/[email protected]/

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 16 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 7 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Quoted 6 lines
Quoted 2 lines

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

For the record:

https://lore.kernel.org/kvm/[email protected]/

Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#7 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #4 from Sean Christopherson ([email protected]) ---

Quoted 24 lines · On Tue, Jun 28, 2022, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 10 lines
Quoted 9 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Quoted 6 lines
Quoted 3 lines

For the record:

https://lore.kernel.org/kvm/[email protected]/

Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?

No, AFAIK no one has posted a fix. If it's the KVM issue I'm thinking of, the fix is non-trivial. It'd require scheduling a timer in L0 with a deadline shorter
than what L1 requests when emulating the VMX timer, and then busy waiting in L0 if
the host timer fires early. KVM already does this for e.g. L1's TSC deadline timer.
That code would need to be adapated for the nested VMX preemption timer.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw

Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]
Quoted 27 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 19 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 9 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Quoted 6 lines
Quoted 3 lines

For the record:

https://lore.kernel.org/kvm/[email protected]/

Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?

As I noted, at some point I did not manage to reproduce the failure.

The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.

#8 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #5 from Nadav Amit ([email protected]) ---

Quoted 25 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 19 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 9 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Quoted 6 lines
Quoted 3 lines

For the record:

https://lore.kernel.org/kvm/[email protected]/

Thanks for your reply. So this is a KVM bug, and you have sent a patch to kvm to fix this bug, right?

As I noted, at some point I did not manage to reproduce the failure.

The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#9 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #6 from Sean Christopherson ([email protected]) ---

Quoted 36 lines · On Tue, Jun 28, 2022, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #5 from Nadav Amit ([email protected]) ---

Quoted 15 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky (though IIRC
it's KVM that's at fault).

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

For the record:

https://lore.kernel.org/kvm/[email protected]/

Quoted 2 lines

Thanks for your reply. So this is a KVM bug, and you have sent a patch to

kvm

Quoted 8 lines
Quoted 1 line

to fix this bug, right?

As I noted, at some point I did not manage to reproduce the failure.

The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be
a KVM bug.

Oooh, your failure was on bare-metal. I didn't grok that. Though it could be both a hardware bug and a KVM bug :-)

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#10 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #7 from Yang Lixiao ([email protected]) --- (In reply to Sean Christopherson from comment #6)

Quoted 45 lines · On Tue, Jun 28, 2022, [email protected] wrote:
Quoted 19 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #5 from Nadav Amit ([email protected]) ---

Quoted 15 lines · On Jun 27, 2022, at 6:19 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #3 from Yang Lixiao ([email protected]) --- (In reply to Nadav Amit from comment #2)

Quoted 9 lines
Quoted 8 lines · On Jun 27, 2022, at 5:28 PM, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Sean Christopherson ([email protected]) changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

CC| |[email protected]

--- Comment #1 from Sean Christopherson ([email protected]) --- It's vmx_preemption_timer_expiry_test, which is known to be flaky

(though

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

IIRC
it's KVM that's at fault).

Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

For the record:

https://lore.kernel.org/kvm/[email protected]/

Quoted 1 line
Quoted 7 lines
Quoted 1 line

Thanks for your reply. So this is a KVM bug, and you have sent a patch to

kvm

Quoted 1 line

to fix this bug, right?

As I noted, at some point I did not manage to reproduce the failure.

The failure on bare-metal that I experienced hints that this is either a

test

Quoted 8 lines
Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is likely

to

Quoted 2 lines

be
a KVM bug.

Oooh, your failure was on bare-metal. I didn't grok that. Though it could be
both a hardware bug and a KVM bug :-)

In my tests, I tested kvm-unit-tests vmx on bare-metal (not on VM) and this bug happened on two different Ice Lake machines and one Cooper Lake machine.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw

Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to Nadav Amit
Quoted 3 lines · Nadav Amit

The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

#11 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #8 from Jim Mattson ([email protected]) ---

Quoted 4 lines · Nadav Amit

The failure on bare-metal that I experienced hints that this is either a test bug or (much less likely) a hardware bug. But I do not think it is likely to be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#12 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 31 lines · Nadav Amit
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is either a

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is likely

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#13 lore raw

Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

Quoted 45 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 31 lines · Nadav Amit
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is either a

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is likely

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Quoted 7 lines

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?

#14 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #10 from Jim Mattson ([email protected]) ---

Quoted 45 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 31 lines · Nadav Amit
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is either a

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is likely

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

Quoted 7 lines

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#15 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)

Quoted 52 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 3 lines · Nadav Amit
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is either

a

Quoted 2 lines
Quoted 2 lines

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is

likely

Quoted 39 lines
Quoted 26 lines

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

Quoted 5 lines
Quoted 3 lines

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?

All of the values of MSR 0x9B on the three hosts are 0.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#16 lore raw

Re: [Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

Quoted 63 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)

Quoted 52 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 3 lines · Nadav Amit
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is either

a

Quoted 2 lines
Quoted 2 lines

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is

likely

Quoted 39 lines
Quoted 26 lines

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

Quoted 5 lines
Quoted 3 lines

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these hosts?

Quoted 7 lines

All of the values of MSR 0x9B on the three hosts are 0.

--
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Doh! There is a glaring bug in the test. I'll post a fix soon.

#17 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #12 from Jim Mattson ([email protected]) ---

Quoted 66 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)

Quoted 7 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 1 line

On Mon, Jun 27, 2022 at 8:54 PM Nadav Amit <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is

either

Quoted 43 lines

a

Quoted 2 lines
Quoted 2 lines

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is

likely

Quoted 39 lines
Quoted 26 lines

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the test assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Quoted 4 lines

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

Quoted 5 lines
Quoted 3 lines

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these

hosts?

Quoted 7 lines

All of the values of MSR 0x9B on the three hosts are 0.

--
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Doh! There is a glaring bug in the test. I'll post a fix soon.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#18 lore raw

[Bug 216177] kvm-unit-tests vmx has about 60% of failure chance

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #13 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #12)

Quoted 68 lines · On Tue, Jun 28, 2022 at 5:22 PM <[email protected]> wrote:
Quoted 9 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

--- Comment #11 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #10)

Quoted 4 lines · On Mon, Jun 27, 2022 at 11:32 PM <[email protected]> wrote:
Quoted 3 lines

https://bugzilla.kernel.org/show_bug.cgi?id=216177

Quoted 32 lines
Quoted 3 lines
Quoted 3 lines

--- Comment #9 from Yang Lixiao ([email protected]) --- (In reply to Jim Mattson from comment #8)

Quoted 1 line

On Mon, Jun 27, 2022 at 8:54 PM Nadav Amit <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The failure on bare-metal that I experienced hints that this is

either

Quoted 25 lines

a

Quoted 2 lines
Quoted 2 lines

test

Quoted 1 line

bug or (much less likely) a hardware bug. But I do not think it is

likely

Quoted 21 lines
Quoted 21 lines

to

Quoted 2 lines

be
a KVM bug.

KVM does not use the VMX-preemption timer to virtualize L1's VMX-preemption timer (and that is why KVM is broken). The KVM bug was introduced with commit f4124500c2c1 ("KVM: nVMX: Fully emulate preemption timer"), which uses an L0 CLOCK_MONOTONIC hrtimer to emulate L1's VMX-preemption timer. There are many reasons that this cannot possibly work, not the least of which is that the CLOCK_MONOTONIC timer is subject to time slew.

Currently, KVM reserves L0's VMX-preemption timer for emulating L1's APIC timer. Better would be to determine whether L1's APIC timer or L1's VMX-preemption timer is scheduled to fire first, and use L0's VMX-preemption timer to trigger a VM-exit on the nearest alarm. Alternatively, as Sean noted, one could perhaps arrange for the hrtimer to fire early enough that it won't fire late, but I don't really think that's a viable solution.

I can't explain the bare-metal failures, but I will note that the

test

Quoted 25 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines

assumes the default treatment of SMIs and SMM. The test will likely fail with the dual-monitor treatment of SMIs and SMM. Aside from the older CPUs with broken VMX-preemption timers, I don't know of any relevant errata.

Of course, it is possible that the test itself is buggy. For the person who reported bare-metal failures on Ice Lake and Cooper Lake, how long was the test in VMX non-root mode past the VMX-preemption timer deadline?

On the first Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (28067103426) < TSC deadline (28067086048)

On the second Ice lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (27014488614) < TSC deadline (27014469152)

Quoted 5 lines
Quoted 3 lines

On Cooper lake:
Test suite: vmx_preemption_timer_expiry_test
FAIL: Last stored guest TSC (29030585690) < TSC deadline (29030565024)

Wow! Those are *huge* overruns. What is the value of MSR 0x9B on these

hosts?

Quoted 8 lines
Quoted 7 lines

All of the values of MSR 0x9B on the three hosts are 0.

--
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Doh! There is a glaring bug in the test. I'll post a fix soon.

Thanks!

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.