soft lockup after stop/cont

DISCUSSION Jim Paris 5 participants 14 messages Oct 10, 2007 – Oct 11, 2007 12 messages not in the archive KVM

Share & tools

Jim Paris

author

2007-10-10 08:01 UTC #0 lore raw

If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior? What isn't virtualized that allows it to detect that? The host is a core 2 duo.

I have bigger problems in the guest after migrating to a file and restoring from that, but this seems like a good place to start..

-jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

A message that is not in the archive (sent off-list or not yet received): [email protected]

Avi Kivity

2007-10-10 08:04 UTC #2 lore raw

Quoted 3 lines · Jim Paris

If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?

No.

Quoted 4 lines

What isn't virtualized that allows it to
detect that? The host is a core 2 duo.

It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

A message that is not in the archive (sent off-list or not yet received): 470C87AB.6050900-atKUWr5tajBWk0Htik3J/[email protected]

Dong, Eddie

2007-10-10 08:50 UTC #4 lore raw

Quoted 5 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:

It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?

In Xen, we decide to froze the guest time after save/restore. In this way the guest see contiguous timer and thus avoid this issue.

Probably we should do similar here, comments?
thx,eddie

A message that is not in the archive (sent off-list or not yet received): 10EA09EFD8728347A513008B6B0DA77A023643BD-wq7ZOvIWXbNpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org

Avi Kivity

2007-10-10 09:09 UTC #6 lore raw

Quoted 7 lines · Eddie

Quoted 5 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?

Quoted 5 lines

In Xen, we decide to froze the guest time after save/restore. In this way the guest see contiguous timer and thus avoid this issue.
Probably we should do similar here, comments?

This is pause/resume, not save/restore. I think save/restore now has contiguous lapic timer and discontiguous real time.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

A message that is not in the archive (sent off-list or not yet received): 470C96B2.3080905-atKUWr5tajBWk0Htik3J/[email protected]

Dong, Eddie

2007-10-10 09:11 UTC #8 lore raw

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Eddie
Quoted 5 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?

Quoted 9 lines

Quoted 5 lines
In Xen, we decide to froze the guest time after save/restore. In this way the guest see contiguous timer and thus avoid this issue.
Probably we should do similar here, comments?
This is pause/resume, not save/restore. I think save/restore now has contiguous lapic timer and discontiguous real time.

If save/restore means live migration here, yes you are right. save to file and later resume from file will have this issue. thx,eddie

A message that is not in the archive (sent off-list or not yet received): 10EA09EFD8728347A513008B6B0DA77A02364404-wq7ZOvIWXbNpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org

Avi Kivity

2007-10-10 09:18 UTC #10 lore raw

Quoted 19 lines · Eddie

Quoted 17 lines · Avi Kivity
Quoted 15 lines · Eddie
Quoted 6 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?
In Xen, we decide to froze the guest time after save/restore. In this way the guest see contiguous timer and thus avoid this issue.
Probably we should do similar here, comments?

Quoted 7 lines

Quoted 4 lines
This is pause/resume, not save/restore. I think save/restore now has contiguous lapic timer and discontiguous real time.
If save/restore means live migration here, yes you are right.

No, I didn't mean that.

Quoted 2 lines

save to file and later resume from file will have this issue.

Why? after saving to a file, the local apic in the kernel is destroyed when qemu quits.

Or is the timer saved in absolute time? if so you are right, and yes, your solution is needed.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

A message that is not in the archive (sent off-list or not yet received): 470C98EB.1080602-atKUWr5tajBWk0Htik3J/[email protected]

Dong, Eddie

2007-10-10 09:57 UTC #12 lore raw

Quoted 4 lines

Or is the timer saved in absolute time? if so you are right, and yes, your solution is needed.

Looks like current live migration, also save/restore, doesn't migrate guest time. (do I miss something?) So the new guest will see a totally different TSC and OS feel stranger or
many lost ticks etc.

Should we add this one?
thx,eddie

A message that is not in the archive (sent off-list or not yet received): 10EA09EFD8728347A513008B6B0DA77A0236442D-wq7ZOvIWXbNpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org

Avi Kivity

2007-10-10 10:06 UTC #14 lore raw

Quoted 4 lines · Eddie

Quoted 4 lines
Or is the timer saved in absolute time? if so you are right, and yes, your solution is needed.

Quoted 8 lines

Looks like current live migration, also save/restore, doesn't migrate guest time. (do I miss something?) So the new guest will see a totally different TSC and OS feel stranger or
many lost ticks etc.
Should we add this one?
thx,eddie

cpu_save (qemu/hw/pc.c) has this:

#ifdef USE_KVM
    if (kvm_allowed) {
        for (i = 0; i < NR_IRQ_WORDS ; i++) {
            qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);
        }
        qemu_put_be64s(f, &env->tsc);
    }

#endif

And I think the apic code saves the timer?

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

A message that is not in the archive (sent off-list or not yet received): 470CA429.2080607-atKUWr5tajBWk0Htik3J/[email protected]

Dong, Eddie

2007-10-11 01:17 UTC #16 lore raw

Quoted 9 lines · Avi Kivity

cpu_save (qemu/hw/pc.c) has this:

#ifdef USE_KVM
   if (kvm_allowed) {
       for (i = 0; i < NR_IRQ_WORDS ; i++) {
           qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);        }
       qemu_put_be64s(f, &env->tsc);
   }

#endif

Mmm, so this is not the root cause.

Quoted 3 lines

And I think the apic code saves the timer?

APIC timer is restored at kvm_vcpu_ioctl_set_lapic/kvm_apic_post_state_restore/start_apic_timer.

Thx, Eddie

Jim Paris

author

2007-10-10 18:54 UTC #17 lore raw

In reply to a message not in the archive

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Jim Paris
If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?

Quoted 8 lines

No.
Quoted 2 lines
What isn't virtualized that allows it to
detect that? The host is a core 2 duo.
It may be that the timer correction code detects that zillions of timer interrupts have not been serviced by the guest so it floods the guest with these interrupts. Does -no-kvm-irqchip help?

I can't seem to reproduce the "soft lockup" at the moment, but there's a "clocksource tsc unstable" that happens pretty reliably when I pause the VM using stop/cont for about 15 seconds:

$ kvm -M pc -m 256 -smp 1 -boot c -hda test -net none

stop/cont, guest reports:
Clocksource tsc unstable (delta = 15877929568 ns)
Time: acpi_pm clocksource has been installed.

$ kvm -no-kvm-irqchip -M pc -m 256 -smp 1 -boot c -hda test -net none

stop/cont, guest seems not to notice.

(Host is 2.6.20.4, kvm-45 modules & userspace, guest 2.6.21)

-jim

A message that is not in the archive (sent off-list or not yet received): [email protected]

Carsten Otte

2007-10-11 09:03 UTC #19 lore raw

Quoted 3 lines · Jim Paris

If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?

We have the same behavior on s390 when running in a virtual environment. The issue is, that the guest physical cpu may be scheduled away while a timer runs down. I don't think there's anything kvm could do better regerding the soft lockup code. Soft lockup is just broken for virtualized environments and needs to be fixed. For the time being, just disable it in kconfig.

A message that is not in the archive (sent off-list or not yet received): [email protected]

Dong, Eddie

2007-10-11 09:29 UTC #21 lore raw

Quoted 11 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:

Quoted 3 lines · Jim Paris
If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?
We have the same behavior on s390 when running in a virtual environment. The issue is, that the guest physical cpu may be scheduled away while a timer runs down. I don't think there's anything kvm could do better regerding the soft lockup code. Soft lockup is just broken for virtualized environments and needs to be fixed. For the time being, just disable it in kconfig.

This is a well known issue, but we can fix by syncing TSC with PIT etc. But doing this we need to froze guest TSC when a VP is descheduled.

We enable this in Xen but feel it is too complicated so never wanna to do same thing in KVM per previous discussion. There are some side effect of course due to this.

In Xen some time you will see PV domain soft lockup, but never in hardware HVM :-) Hopefully Linux will have pv timer soon.

thx, eddie

A message that is not in the archive (sent off-list or not yet received): 10EA09EFD8728347A513008B6B0DA77A02364AD6-wq7ZOvIWXbNpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org

Carsten Otte

2007-10-11 10:57 UTC #23 lore raw

Quoted 15 lines · Eddie

Quoted 11 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
Quoted 3 lines · Jim Paris
If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?
We have the same behavior on s390 when running in a virtual environment. The issue is, that the guest physical cpu may be scheduled away while a timer runs down. I don't think there's anything kvm could do better regerding the soft lockup code. Soft lockup is just broken for virtualized environments and needs to be fixed. For the time being, just disable it in kconfig.
This is a well known issue, but we can fix by syncing TSC with PIT etc. But doing this we need to froze guest TSC when a VP is descheduled.

Quoted 6 lines

We enable this in Xen but feel it is too complicated so never wanna to do same thing in KVM per previous discussion. There are some side effect of course due to this.
In Xen some time you will see PV domain soft lockup, but never in hardware HVM :-) Hopefully Linux will have pv timer soon.

Yes, virtual timers will be the right fix for it. I believe Jan has a patch for that. Am I mistaken Jan? [adding Jan to CC:]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Jan Glauber1

2007-10-11 16:18 UTC #25 lore raw

Carsten Otte/Germany/IBM@IBMDE wrote on 10/11/2007 12:57:38 PM:

Quoted 16 lines

Quoted 15 lines · Eddie
Quoted 11 lines · kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
Quoted 3 lines · Jim Paris
If I stop KVM in the monitor with "stop", wait a minute, and do "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0". Is that expected behavior?
We have the same behavior on s390 when running in a virtual environment. The issue is, that the guest physical cpu may be scheduled away while a timer runs down. I don't think there's anything kvm could do better regerding the soft lockup code. Soft lockup is just broken for virtualized environments and needs to be fixed. For the time being, just disable it in kconfig.
This is a well known issue, but we can fix by syncing TSC with PIT etc. But doing this we need to froze guest TSC when a VP is descheduled.

Quoted 8 lines

Quoted 6 lines
We enable this in Xen but feel it is too complicated so never wanna to do same thing in KVM per previous discussion. There are some side effect of course due to this.
In Xen some time you will see PV domain soft lockup, but never in hardware HVM :-) Hopefully Linux will have pv timer soon.
Yes, virtual timers will be the right fix for it. I believe Jan has a patch for that. Am I mistaken Jan? [adding Jan to CC:]

Well, we have an instruction for that ;- (but that's not the whole story). The idea is simply to make sched_clock using a virtual timer, so softlockup does only account time the guest was scheduled. Currently we have softlockup disabled
on s390 too, since there are some open issues with CFS and the virtual sched_clock.

-- Jan