New: "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.
https://bugzilla.kernel.org/show_bug.cgi?id=75331
Bug ID: 75331
Summary: "soft lockup CPU#0 stuck for 23s" regression on 32bit
3.13.0+ kernels.
Product: Virtualization
Version: unspecified
Kernel Version: 3.13.0+
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoCreated attachment 134761
--> https://bugzilla.kernel.org/attachment.cgi?id=134761&action=edit
dmesg log and kernel .config
I'm getting "soft lockup CPU#0 stuck for 23s" when running qemu 1.7.0 with kvm on vanilla kernels 3.13.0 and above. All builds are 32 bit.
The lockup happens shortly after starting the guest OS, sometimes even before the guest boot is complete. The cpu graph shows one core running in kernel mode, the guest freezes, qemu is still responsive. Sometimes the lockup goes away after a while, then comes back.
The first time I didn't kill qemu and it managed to lock my system.
I'm not using virtio/xen or other frameworks. Just bare qemu. I've tried to remove network and sound devices but it had no measurable impact. Having -usb and -smp 4 seems to make triggering the lockup much easier/faster but removing them doesn't eliminate lockups. (Or lockups may be random and not related to these options.)
With some help, I started a git bisect on the Linus' repository. I did manage to locate a commit that when reversed eliminates the problem completely (tested on 3.14.1 kernel that previously had this problem).
The commit is :
---------
commit e0f6dec35f9286e78879fe1ac92803fd69fc4fdc
Author: H. Peter Anvin <[email protected]> Date: Wed Dec 4 14:31:28 2013 -0800
x86, bitops: Correct the assembly constraints to testing bitops
---------
I'm with gcc-4.8.2 on latest Slackware.
To this bugreport I'm attaching tgz archive with the 3.14.0 kernel .config file and dmesg extracted from the vmcore.
On request I could provide vmcore and vmlinux of 3.14.0 compiled with debug info. (75MB each, when gzipped/stripped by makedumpfile)
Qemu is compiled with: ./configure --disable-strip --audio-drv-list=oss,alsa,sdl --disable-bsd-user --enable-vnc --target-list=i386-softmmu,i386-linux-user
Qemu is started with : qemu -smp 4 -usb -enable-kvm -vga cirrus -net none -ctrl-grab -display sdl -hda guest.qcow2
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.
https://bugzilla.kernel.org/show_bug.cgi?id=75331
Ivan Kalvachev <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] Hardware|All |i386 Regression|No |Yes
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=75331
--- Comment #1 from Ivan Kalvachev <[email protected]> ---
I tried to isolate the difference in .o object files when compiling the kernel with and without the above commit. Unfortunately there were no differences (only in version, boot and init code).
Upon further testing the kernel with the reverted commit does seem to experience the same lockup, it just took it more time.
The above commit is not the culprit. I'll try to redo the bisect. Sorry for the noise.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=75331
Roland Kletzing ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #2 from Roland Kletzing ([email protected]) ---
this may be caused by intense IO on the host if your VM is not using virtio dataplance.
i have seen such when copying data or live migrating virtual machines
have a look at
https://bugzilla.kernel.org/show_bug.cgi?id=199727 and
https://gitlab.com/qemu-project/qemu/-/issues/819
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=75331
--- Comment #3 from Ivan Kalvachev ([email protected]) --- (In reply to Roland Kletzing from comment #2)
Quoted 2 lines
this may be caused by intense IO on the host if your VM is not using virtio dataplance.
Nothing intense was running at the time.
AFAIR It could happen when both guest&host were mostly idle.
The issue was also completely gone when using 64bit kernel/Slackware and the same 32bit guest. I don't remember if I've tried the same qemu binary.
Unfortunately, I no longer have the kernel memory image dump.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.