Skip to contentSkip to conversation

kvm/x86: perf: Softlockup issue

DISCUSSION Breno Leitao 2 participants 3 messages Oct 3, 2023 – Oct 4, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've been pursuing a bug in a virtual machine (KVM) that I would like to share in here. The VM gets stuck when running perf in a VM and getting soft lockups.

The bug happens upstream (Linux 6.6-rc4 - 8a749fd1a8720d461). The same kernel is being used in the host and in the guest.

The problem only happens in a very specific circumstances:

1) PMU needs to be enabled in the guest

2) Libvirt/QEMU needs to use a custom CPU:
* Here is the qemu line:
-cpu Skylake-Server,kvm-pv-eoi=on,pmu=on
* Any other CPU seems to hit the problem
* Even using Skylake-Server on a Skylake server
* Using CPU passthrough workaround the problem

3) You need to use 6 or more events in perf.
* This is a line that reproduces the problem:
# perf stat -e cpu-clock -e context-switches -e cpu-migrations -e page-faults -e cycles -e instructions -e branches ls
* Removing any of these events (totaling 5 events) makes perf work again

4) This problem happens on upstream, 6.4 and 5.19
* This problem doesn't seem to happen on 5.12

Problem
========

When running perf in the circumstances above, the VM is stuck, with a lot of stack traces. This is some messages:

	 kernel:[  400.314381] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/u68:11:6853]
	 kernel:[  400.324380] watchdog: BUG: soft lockup - CPU#8 stuck for 26s! [dynoKernelMon:9781]
	 kernel:[  404.368380] watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kworker/30:2:1326]

Here is part of the stack. The full stack is in the pastebin below:

	 nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
	 nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47)
	 nmi_handle (arch/x86/kernel/nmi.c:149)
	 __intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147 arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
	 __intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147 arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
	 default_do_nmi (arch/x86/kernel/nmi.c:347)
	 exc_nmi (arch/x86/kernel/nmi.c:543)
	 end_repeat_nmi (arch/x86/entry/entry_64.S:1471)
	 __intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147 arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
	 __intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147 arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
	 __intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147 arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)

More info
=========

Soft lockup messages in the guest:

	https://paste.debian.net/1293888/
Full log from the guest:
	https://paste.debian.net/1293891/
vCPU stacks dumped from the host (cat /proc/<vcpu>/stack):
	https://paste.debian.net/1293887/
Qemu (version 7.1.0) command line
	https://paste.debian.net/1293894/
#1 lore raw
Quoted 6 lines · Breno Leitao

I've been pursuing a bug in a virtual machine (KVM) that I would like to share in here. The VM gets stuck when running perf in a VM and getting soft lockups.

The bug happens upstream (Linux 6.6-rc4 - 8a749fd1a8720d461). The same kernel is being used in the host and in the guest.

Have you tried https://lore.kernel.org/kvm/[email protected]/?

author Tested-by
#2 lore raw

Hello Jim,

Quoted 9 lines · Jim Mattson
Quoted 6 lines · Breno Leitao

I've been pursuing a bug in a virtual machine (KVM) that I would like to share in here. The VM gets stuck when running perf in a VM and getting soft lockups.

The bug happens upstream (Linux 6.6-rc4 - 8a749fd1a8720d461). The same kernel is being used in the host and in the guest.

Have you tried https://lore.kernel.org/kvm/[email protected]/?

Thanks for the heads-up. These two patches indeed fix the problem. Thanks for getting the problem fixed.

Breno