Skip to contentSkip to conversation

KVM Linux guest with more than 1 CPU panics after commit 5a9f54435a48... on old CPU (Phenom x4)

Bug 209155 [email protected] 2 participants 8 messages Sep 4, 2020 – Apr 21 1 message not in the archive KVM

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

--- Comment #3 from Paul K. ([email protected]) ---

$ git bisect log
# bad: [bcf876870b95592b52519ed4aafcf9d95999bc9c] Linux 5.8
git bisect start 'v5.8'
# good: [3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162] Linux 5.7
git bisect good 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162
# bad: [694b5a5d313f3997764b67d52bab66ec7e59e714] Merge tag 'arm-soc-5.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 694b5a5d313f3997764b67d52bab66ec7e59e714
# bad: [694b5a5d313f3997764b67d52bab66ec7e59e714] Merge tag 'arm-soc-5.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 694b5a5d313f3997764b67d52bab66ec7e59e714
# bad: [694b5a5d313f3997764b67d52bab66ec7e59e714] Merge tag 'arm-soc-5.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 694b5a5d313f3997764b67d52bab66ec7e59e714
# bad: [694b5a5d313f3997764b67d52bab66ec7e59e714] Merge tag 'arm-soc-5.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 694b5a5d313f3997764b67d52bab66ec7e59e714
# bad: [694b5a5d313f3997764b67d52bab66ec7e59e714] Merge tag 'arm-soc-5.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 694b5a5d313f3997764b67d52bab66ec7e59e714
# bad: [2e63f6ce7ed2c4ff83ba30ad9ccad422289a6c63] Merge branch 'uaccess.comedi'
of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 2e63f6ce7ed2c4ff83ba30ad9ccad422289a6c63
# good: [cfa3b8068b09f25037146bfd5eed041b78878bee] Merge tag 'for-linus-hmm' of
git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
git bisect good cfa3b8068b09f25037146bfd5eed041b78878bee
# good: [c41219fda6e04255c44d37fd2c0d898c1c46abf1] Merge tag
'drm-intel-next-fixes-2020-05-20' of
git://anongit.freedesktop.org/drm/drm-intel into drm-next
git bisect good c41219fda6e04255c44d37fd2c0d898c1c46abf1
# good: [f3cdc8ae116e27d84e1f33c7a2995960cebb73ac] Merge tag 'for-5.8-tag' of
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
git bisect good f3cdc8ae116e27d84e1f33c7a2995960cebb73ac
# good: [f1e455352b6f503532eb3637d0a6d991895e7856] Merge tag 'kgdb-5.8-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/danielt/linux
git bisect good f1e455352b6f503532eb3637d0a6d991895e7856
# bad: [cb953129bfe5c0f2da835a0469930873fb7e71df] kvm: add halt-polling cpu
usage stats
git bisect bad cb953129bfe5c0f2da835a0469930873fb7e71df
# good: [3754afe7cf7cc3693a9c9ff795e9bd97175ca639] tools/kvm_stat: Add command
line switch '-L' to log to file
git bisect good 3754afe7cf7cc3693a9c9ff795e9bd97175ca639
# good: [c4e115f08c08cb9f3b70247b42323e40b9afd1fd] kvm/eventfd: remove unneeded
conversion to bool
git bisect good c4e115f08c08cb9f3b70247b42323e40b9afd1fd
# good: [5b494aea13fe9ec67365510c0d75835428cbb303] KVM: No need to retry for
hva_to_pfn_remapped()
git bisect good 5b494aea13fe9ec67365510c0d75835428cbb303
# bad: [379a3c8ee44440d5afa505230ed8cb5b0d0e314b] KVM: VMX: Optimize
posted-interrupt delivery for timer fastpath
git bisect bad 379a3c8ee44440d5afa505230ed8cb5b0d0e314b
# good: [9e826feb8f114964cbdce026340b6cb9bde68a18] KVM: nVMX: Drop superfluous
VMREAD of vmcs02.GUEST_SYSENTER_*
git bisect good 9e826feb8f114964cbdce026340b6cb9bde68a18
# good: [2c4c41325540cf3abb12aef142c0e550f6afeffc] KVM: x86: Print symbolic
names of VMX VM-Exit flags in traces
git bisect good 2c4c41325540cf3abb12aef142c0e550f6afeffc
# bad: [404d5d7bff0d419fe11c7eaebca9ec8f25258f95] KVM: X86: Introduce more
exit_fastpath_completion enum values
git bisect bad 404d5d7bff0d419fe11c7eaebca9ec8f25258f95
# good: [5a9f54435a488f8a1153efd36cccee3e7e0fc28b] KVM: X86: Introduce
kvm_vcpu_exit_request() helper
git bisect good 5a9f54435a488f8a1153efd36cccee3e7e0fc28b
# first bad commit: [404d5d7bff0d419fe11c7eaebca9ec8f25258f95] KVM: X86:
Introduce more exit_fastpath_completion enum values
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

Wanpeng Li ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #4 from Wanpeng Li ([email protected]) ---
Could you dump the lscpu results in both the guest and the host? In addition, could you dump the result from grep . /sys/module/kvm_amd/parameters/* ?

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |sean.j.christopherson@intel
                   |                            |.com

--- Comment #8 from Sean Christopherson ([email protected]) --- From code inspection, I'm 99% confident the immediate bug is that svm->next_rip is reset in svm_vcpu_run() only after calling svm_exit_handlers_fastpath(), which will cause SVM's skip_emulated_instruction() to write a stale RIP. I don't have AMD hardware to confirm, but this should be reproducible on modern CPUs by loading kvm_amd with nrips=0.

That issue is easy enough to resolve, e.g. simply hoist "svm->next_rip = 0;" up above the fastpath handling. But, there are additional complications with advancing rip in the fastpath as svm_complete_interrupts() consumes rip, e.g. for NMI unmasking logic and event reinjection. Odds are that NMI unmasking will never "fail" as it would require the new rip to match the last IRET rip, which would be very bizarre. Similarly, event reinjection should also be a non-issue in practice as the WRMSR fastpath shouldn't be reachable if KVM was injecting an event.

All the being said, IMO, the safest play would be to first yank out the call to handle_fastpath_set_msr_irqoff() in svm_exit_handlers_fastpath() to ensure a clean base and to provide a safe backport patch, then move svm_complete_interrupts() into svm_vcpu_run(), and finally move the call to svm_exit_handlers_fastpath() down a ways and reenable handle_fastpath_set_msr_irqoff(). Aside from resolving weirdness with rip and fastpath, it would also align VMX and SVM with respect to completing interrupts.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

--- Comment #11 from Wanpeng Li ([email protected]) --- (In reply to Sean Christopherson from comment #8)

Quoted 16 lines

From code inspection, I'm 99% confident the immediate bug is that svm->next_rip is reset in svm_vcpu_run() only after calling svm_exit_handlers_fastpath(), which will cause SVM's skip_emulated_instruction() to write a stale RIP. I don't have AMD hardware to confirm, but this should be reproducible on modern CPUs by loading kvm_amd with nrips=0.

That issue is easy enough to resolve, e.g. simply hoist "svm->next_rip = 0;" up above the fastpath handling. But, there are additional complications with advancing rip in the fastpath as svm_complete_interrupts() consumes rip, e.g. for NMI unmasking logic and event reinjection. Odds are that NMI unmasking will never "fail" as it would require the new rip to match the last IRET rip, which would be very bizarre. Similarly, event reinjection should also be a non-issue in practice as the WRMSR fastpath shouldn't be reachable if KVM was injecting an event.

Quoted 8 lines

All the being said, IMO, the safest play would be to first yank out the call to handle_fastpath_set_msr_irqoff() in svm_exit_handlers_fastpath() to ensure a clean base and to provide a safe backport patch, then move svm_complete_interrupts() into svm_vcpu_run(), and finally move the call to svm_exit_handlers_fastpath() down a ways and reenable handle_fastpath_set_msr_irqoff(). Aside from resolving weirdness with rip and fastpath, it would also align VMX and SVM with respect to completing interrupts.

Hi Sean, thanks for your analyses, I will send out patches to fix it. :)

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

--- Comment #12 from Paul K. ([email protected]) --- Verified fix works in both the bisected revision and v5.9-rc3. I have not tried to apply the three patches sent to the mailing list. Should I?

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

--- Comment #13 from Wanpeng Li ([email protected]) --- (In reply to Paul K. from comment #12)

Quoted 2 lines

Verified fix works in both the bisected revision and v5.9-rc3. I have not tried to apply the three patches sent to the mailing list. Should I?

Please have a try, we test these three patches on AMD ROME with nrips=0.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#7 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

--- Comment #14 from Paul K. ([email protected]) --- I'm having a bit of a problem applying the patches cleanly. Working with both v5.9-rc3 and 5.9-rc4 give the same:

Patch 1/3 goes fine:

$ patch -p1 < /net/phenom/export/home2/users/kronenpj/tmp/patch1-3.txt 
patching file arch/x86/kvm/svm/svm.c

Patch 2/3 fails on hunk #3:

$ patch -p1 < /net/phenom/export/home2/users/kronenpj/tmp/patch2-3.txt 
patching file arch/x86/kvm/svm/svm.c
Hunk #1 succeeded at 3349 (offset 2 lines).
Hunk #2 succeeded at 3504 (offset 4 lines).
Hunk #3 FAILED at 3533.
1 out of 3 hunks FAILED -- saving rejects to file arch/x86/kvm/svm/svm.c.rej
$ cat svm.c.rej 
--- arch/x86/kvm/svm/svm.c
+++ arch/x86/kvm/svm/svm.c
@@ -3533,6 +3537,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu
*vcpu)
                svm_handle_mce(svm);
        svm_complete_interrupts(svm);

+ exit_fastpath = svm_exit_handlers_fastpath(vcpu);

        vmcb_mark_all_clean(svm->vmcb);
        return exit_fastpath;

Adding that line manually and continuing with the third patch:

$ patch -p1 < /net/phenom/export/home2/users/kronenpj/tmp/patch3-3.txt 
patching file arch/x86/kvm/svm/svm.c
Hunk #2 succeeded at 3536 with fuzz 2 (offset 8 lines).

The patch against v5.9-rc4+ works as expected.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#8 lore raw

[Bug 209155] KVM Linux guest with more than 1 CPU panics after commit 404d5d7bff0d419fe11c7eaebca9ec8f25258f95 on old CPU (Phenom x4)

In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=209155

Paul K. ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |OBSOLETE
Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.