Skip to contentSkip to conversation

New: "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

Bug 75331 [email protected] 2 participants 5 messages May 2, 2014 – Feb 8, 2023 KVM

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=75331

            Bug ID: 75331
           Summary: "soft lockup CPU#0 stuck for 23s" regression on 32bit
                    3.13.0+ kernels.
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.13.0+
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 134761
--> https://bugzilla.kernel.org/attachment.cgi?id=134761&action=edit
dmesg log and kernel .config

I'm getting "soft lockup CPU#0 stuck for 23s" when running qemu 1.7.0 with kvm on vanilla kernels 3.13.0 and above. All builds are 32 bit.

The lockup happens shortly after starting the guest OS, sometimes even before the guest boot is complete. The cpu graph shows one core running in kernel mode, the guest freezes, qemu is still responsive. Sometimes the lockup goes away after a while, then comes back.
The first time I didn't kill qemu and it managed to lock my system.

I'm not using virtio/xen or other frameworks. Just bare qemu. I've tried to remove network and sound devices but it had no measurable impact. Having -usb and -smp 4 seems to make triggering the lockup much easier/faster but removing them doesn't eliminate lockups. (Or lockups may be random and not related to these options.)

With some help, I started a git bisect on the Linus' repository. I did manage to locate a commit that when reversed eliminates the problem completely (tested on 3.14.1 kernel that previously had this problem).
The commit is :
---------
commit e0f6dec35f9286e78879fe1ac92803fd69fc4fdc

Author: H. Peter Anvin <[email protected]>
Date:   Wed Dec 4 14:31:28 2013 -0800

x86, bitops: Correct the assembly constraints to testing bitops
---------

I'm with gcc-4.8.2 on latest Slackware.

To this bugreport I'm attaching tgz archive with the 3.14.0 kernel .config file and dmesg extracted from the vmcore.

On request I could provide vmcore and vmlinux of 3.14.0 compiled with debug info. (75MB each, when gzipped/stripped by makedumpfile)

Qemu is compiled with: ./configure --disable-strip --audio-drv-list=oss,alsa,sdl --disable-bsd-user --enable-vnc --target-list=i386-softmmu,i386-linux-user

Qemu is started with : qemu -smp 4 -usb -enable-kvm -vga cirrus -net none -ctrl-grab -display sdl -hda guest.qcow2

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

--- Comment #1 from Ivan Kalvachev <[email protected]> ---
I tried to isolate the difference in .o object files when compiling the kernel with and without the above commit. Unfortunately there were no differences (only in version, boot and init code).

Upon further testing the kernel with the reverted commit does seem to experience the same lockup, it just took it more time.

The above commit is not the culprit. I'll try to redo the bisect. Sorry for the noise.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

Roland Kletzing ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Roland Kletzing ([email protected]) ---
this may be caused by intense IO on the host if your VM is not using virtio dataplance.

i have seen such when copying data or live migrating virtual machines

have a look at
https://bugzilla.kernel.org/show_bug.cgi?id=199727 and
https://gitlab.com/qemu-project/qemu/-/issues/819

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

--- Comment #3 from Ivan Kalvachev ([email protected]) --- (In reply to Roland Kletzing from comment #2)

Quoted 2 lines

this may be caused by intense IO on the host if your VM is not using virtio dataplance.

Nothing intense was running at the time.
AFAIR It could happen when both guest&host were mostly idle.

The issue was also completely gone when using 64bit kernel/Slackware and the same 32bit guest. I don't remember if I've tried the same qemu binary.

Unfortunately, I no longer have the kernel memory image dump.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.