Skip to contentSkip to conversation

Soft lockups and cpu frequency scaling

DISCUSSION Martin Schmitt 2 participants 7 messages Dec 22, 2009 – Jan 6, 2010 KVM

lore
Share & tools
lore

Martin Schmitt

author
#0 lore raw

All,

I'm running a manually compiled KVM on CentOS 5.4. The KVM installation has been carried over from CentOS 5.3, when KVM wasn't distributed with the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't able to get along with RedHat's interpretation of KVM.)

The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't seem to compile on CentOS' current 2.6.18-164.9.1.el5.

Only on CentOS guests, I see very frequent "soft lockup" messages and excessively hanging KVM instances. Long-term stability is fine (several months uptime), but disturbed by the hangs. The problem already was there on CentOS 5.3 as well. With the Debian guests on the same host, I have never had any apparent problems.

A number of google results suggest that I should work with CPU scaling on the CentOS guest systems, but unfortunately, CPU scaling is not available in my guests. So, here's my question: How do I enable CPU scaling in KVM guests? Or is there any other measure against these soft lockups that you can recommend?

Thanks everyone,

-martin

Signature · 2 lines
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--
#1 lore raw
Quoted 7 lines · Martin Schmitt

All,

I'm running a manually compiled KVM on CentOS 5.4. The KVM installation has been carried over from CentOS 5.3, when KVM wasn't distributed with the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't able to get along with RedHat's interpretation of KVM.)

Quoted 5 lines

The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't seem to compile on CentOS' current 2.6.18-164.9.1.el5.

Only on CentOS guests, I see very frequent "soft lockup" messages and excessively hanging KVM instances.

Can you please share some of the soft lockup messages.

And how exactly are the VMs hanging?

Quoted 4 lines

Long-term stability is fine (several months uptime), but disturbed
by the hangs. The problem already was there on CentOS 5.3 as well. With the Debian guests on the same host, I have never had any apparent problems.

Questions:

- Is there significant swapping on the host?
- Are you migrating vm's?

Quoted 5 lines

A number of google results suggest that I should work with CPU scaling on the CentOS guest systems, but unfortunately, CPU scaling is not available in my guests. So, here's my question: How do I enable CPU scaling in KVM guests? Or is there any other measure against these soft lockups that you can recommend?

What probably was suggested is to disable cpu frequency scaling on the host. Please provide more details on the host system.

author
#2 lore raw

Ciao Marcelo,

sorry for getting back so late. Thanks for your patience. :-)

Quoted 8 lines · Marcelo Tosatti
Quoted 8 lines

I'm running a manually compiled KVM on CentOS 5.4. The KVM installation has been carried over from CentOS 5.3, when KVM wasn't distributed with the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't able to get along with RedHat's interpretation of KVM.)

The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't seem to compile on CentOS' current 2.6.18-164.9.1.el5.

Quoted 6 lines
Quoted 2 lines

Only on CentOS guests, I see very frequent "soft lockup" messages and excessively hanging KVM instances.

Can you please share some of the soft lockup messages.

And how exactly are the VMs hanging?

They are unresponsive for a few seconds. More "hiccuping" than hanging. It appears to be I/O-related in some way, because it happens most frequently when I do things on the file system.

Dmesg is full of these:

BUG: soft lockup - CPU#0 stuck for 10s! [kblockd/0:10]

Pid: 10, comm: kblockd/0
EIP: 0060:[<c056f931>] CPU: 0
EIP is at ide_outb+0x4/0x5
EFLAGS: 00000202 Not tainted (2.6.18-164.6.1.el5 #1)
EAX: 00000001 EBX: c07e2f80 ECX: 00000286 EDX: 0000c000
ESI: 00000011 EDI: 00000000 EBP: c07e3014 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f3c000 CR3: 12122000 CR4: 000006d0

 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04de1b9>] freed_request+0x1d/0x37
 [<c056d8d0>] ide_end_request+0xcc/0xd4
 [<c056e221>] ide_intr+0x167/0x190
 [<c044da39>] handle_IRQ_event+0x45/0x8c
 [<c044db04>] __do_IRQ+0x84/0xd6
 [<c044da80>] __do_IRQ+0x0/0xd6
 [<c04074b2>] do_IRQ+0x99/0xc3
 [<c0405946>] common_interrupt+0x1a/0x20
 [<c04291ab>] __do_softirq+0x57/0x114
 [<c04073cf>] do_softirq+0x52/0x9c
 [<c04059d7>] apic_timer_interrupt+0x1f/0x24
 [<c056f931>] ide_outb+0x4/0x5
 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04e710f>] cfq_kick_queue+0x70/0x80
 [<c0431e8a>] run_workqueue+0x78/0xb5
 [<c04e709f>] cfq_kick_queue+0x0/0x80
 [<c043273e>] worker_thread+0xd9/0x10b
 [<c041e727>] default_wake_function+0x0/0xc
 [<c0432665>] worker_thread+0x0/0x10b
 [<c0434b55>] kthread+0xc0/0xeb
 [<c0434a95>] kthread+0x0/0xeb
 [<c0405c53>] kernel_thread_helper+0x7/0x10
 =======================
Quoted 9 lines
Quoted 4 lines

Long-term stability is fine (several months uptime), but disturbed
by the hangs. The problem already was there on CentOS 5.3 as well. With the Debian guests on the same host, I have never had any apparent problems.

Questions:

- Is there significant swapping on the host?
- Are you migrating vm's?

No migration and no swap activity. The host has plenty of idle RAM:

[root@zulu ~]# free -m

             total       used       free     shared    buffers     cached
Mem:          7987       7904         82          0        667       5101
-/+ buffers/cache:       2135       5851

Swap: 1983 0 1983

Quoted 8 lines
Quoted 5 lines

A number of google results suggest that I should work with CPU scaling on the CentOS guest systems, but unfortunately, CPU scaling is not available in my guests. So, here's my question: How do I enable CPU scaling in KVM guests? Or is there any other measure against these soft lockups that you can recommend?

What probably was suggested is to disable cpu frequency scaling on the host. Please provide more details on the host system.

Host is a Quadcore Xeon HP DL320 G5 with CentOS 5.4, old Kernel 2.6.18-128.7.1.el5.

There are no hints toward CPU scaling in /sys/devices/system/ on the host:

[root@zulu ~]# ls -l /sys/devices/system/cpu/cpu0
total 0

drwxr-xr-x 5 root root    0 Nov  7 13:47 cache
-r-------- 1 root root 4096 Jan  4 14:55 crash_notes
drwxr-xr-x 2 root root    0 Nov  7 13:48 topology

The file "Crash Notes" contains the following number: 22792b400

Thanks for your help,

-martin

Signature · 5 lines
Martin Schmitt - Schmitt Systemberatung - http://www.scsy.de
DE 35415 Pohlheim, Gießener Str. 18
DE 65307 Bad Schwalbach, Am Bräunchesberg 9
Linux/UNIX - Internet - E-Mail Infrastructure - Antispam/Antivirus
- "What goes up, must come down. Ask any system administrator." -
#3 lore raw
Quoted 97 lines · Martin Schmitt

Ciao Marcelo,

sorry for getting back so late. Thanks for your patience. :-)

Quoted 8 lines · Marcelo Tosatti
Quoted 8 lines

I'm running a manually compiled KVM on CentOS 5.4. The KVM installation has been carried over from CentOS 5.3, when KVM wasn't distributed with the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't able to get along with RedHat's interpretation of KVM.)

The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't seem to compile on CentOS' current 2.6.18-164.9.1.el5.

Quoted 6 lines
Quoted 2 lines

Only on CentOS guests, I see very frequent "soft lockup" messages and excessively hanging KVM instances.

Can you please share some of the soft lockup messages.

And how exactly are the VMs hanging?

They are unresponsive for a few seconds. More "hiccuping" than hanging. It appears to be I/O-related in some way, because it happens most frequently when I do things on the file system.

Dmesg is full of these:

BUG: soft lockup - CPU#0 stuck for 10s! [kblockd/0:10]

Pid: 10, comm: kblockd/0
EIP: 0060:[<c056f931>] CPU: 0
EIP is at ide_outb+0x4/0x5
EFLAGS: 00000202 Not tainted (2.6.18-164.6.1.el5 #1)
EAX: 00000001 EBX: c07e2f80 ECX: 00000286 EDX: 0000c000
ESI: 00000011 EDI: 00000000 EBP: c07e3014 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f3c000 CR3: 12122000 CR4: 000006d0

 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04de1b9>] freed_request+0x1d/0x37
 [<c056d8d0>] ide_end_request+0xcc/0xd4
 [<c056e221>] ide_intr+0x167/0x190
 [<c044da39>] handle_IRQ_event+0x45/0x8c
 [<c044db04>] __do_IRQ+0x84/0xd6
 [<c044da80>] __do_IRQ+0x0/0xd6
 [<c04074b2>] do_IRQ+0x99/0xc3
 [<c0405946>] common_interrupt+0x1a/0x20
 [<c04291ab>] __do_softirq+0x57/0x114
 [<c04073cf>] do_softirq+0x52/0x9c
 [<c04059d7>] apic_timer_interrupt+0x1f/0x24
 [<c056f931>] ide_outb+0x4/0x5
 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04e710f>] cfq_kick_queue+0x70/0x80
 [<c0431e8a>] run_workqueue+0x78/0xb5
 [<c04e709f>] cfq_kick_queue+0x0/0x80
 [<c043273e>] worker_thread+0xd9/0x10b
 [<c041e727>] default_wake_function+0x0/0xc
 [<c0432665>] worker_thread+0x0/0x10b
 [<c0434b55>] kthread+0xc0/0xeb
 [<c0434a95>] kthread+0x0/0xeb
 [<c0405c53>] kernel_thread_helper+0x7/0x10
 =======================
Quoted 8 lines
Quoted 3 lines

by the hangs. The problem already was there on CentOS 5.3 as well. With the Debian guests on the same host, I have never had any apparent problems.

Questions:

- Is there significant swapping on the host?
- Are you migrating vm's?

No migration and no swap activity. The host has plenty of idle RAM:

[root@zulu ~]# free -m

             total       used       free     shared    buffers     cached
Mem:          7987       7904         82          0        667       5101
-/+ buffers/cache:       2135       5851

Swap: 1983 0 1983

Quoted 8 lines
Quoted 5 lines

A number of google results suggest that I should work with CPU scaling on the CentOS guest systems, but unfortunately, CPU scaling is not available in my guests. So, here's my question: How do I enable CPU scaling in KVM guests? Or is there any other measure against these soft lockups that you can recommend?

What probably was suggested is to disable cpu frequency scaling on the host. Please provide more details on the host system.

Host is a Quadcore Xeon HP DL320 G5 with CentOS 5.4, old Kernel 2.6.18-128.7.1.el5.

There are no hints toward CPU scaling in /sys/devices/system/ on the host:

[root@zulu ~]# ls -l /sys/devices/system/cpu/cpu0
total 0

Quoted 7 lines
drwxr-xr-x 5 root root    0 Nov  7 13:47 cache
-r-------- 1 root root 4096 Jan  4 14:55 crash_notes
drwxr-xr-x 2 root root    0 Nov  7 13:48 topology

The file "Crash Notes" contains the following number: 22792b400

Thanks for your help,

Martin,

Can you please share a few more "soft lockup" messages? (with backtrace included).

Also qemu command line.

And boot-up messages of host and guest.

Thanks

author
#4 lore raw
Quoted 2 lines · Marcelo Tosatti

Can you please share a few more "soft lockup" messages? (with backtrace included).

Full dmesg from guest: http://pastebin.com/f51a966df

Quoted 3 lines

Also qemu command line.

From ps:

/usr/local/kvm/bin/qemu-system-x86_64 -hda /drbd/vweb/vweb.vmdk -vnc 127.0.0.1:4 -m 512 -net nic,macaddr=00:16:3e:6a:a5:10 -net tap -enable-kvm -k de -pidfile /drbd/vweb/pidfile.pid -monitor tcp:127.0.0.1:1004,server,nowait -name /drbd/vweb/vweb.vmdk -cpu host -daemonize

Quoted 1 line

And boot-up messages of host and guest.

Does the dmesg above cover that for the guest or are you asking for anything further than that?

Host side dmesg: http://pastebin.com/f47a30d22

Thanks for your time,

-martin

Signature · 2 lines
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--
#5 lore raw
Quoted 9 lines · Martin Schmitt
Quoted 2 lines · Marcelo Tosatti

Can you please share a few more "soft lockup" messages? (with backtrace included).

Full dmesg from guest: http://pastebin.com/f51a966df

Quoted 1 line

Also qemu command line.

Quoted 7 lines
Quoted 1 line

From ps:

/usr/local/kvm/bin/qemu-system-x86_64 -hda /drbd/vweb/vweb.vmdk -vnc 127.0.0.1:4 -m 512 -net nic,macaddr=00:16:3e:6a:a5:10 -net tap -enable-kvm -k de -pidfile /drbd/vweb/pidfile.pid -monitor tcp:127.0.0.1:1004,server,nowait -name /drbd/vweb/vweb.vmdk -cpu host -daemonize

Hum, can you try converting that vmdk image to qcow2 or raw? (with qemu-img convert).

AFAICS the QEMU vmdk implementation is synchronous, so the guest waits on IO operations to complete on the host side.

Quoted 10 lines
Quoted 1 line

And boot-up messages of host and guest.

Does the dmesg above cover that for the guest or are you asking for anything further than that?

Host side dmesg: http://pastebin.com/f47a30d22

Thanks for your time,

-martin

author
#6 lore raw
Quoted 5 lines · Marcelo Tosatti

Hum, can you try converting that vmdk image to qcow2 or raw? (with qemu-img convert).

AFAICS the QEMU vmdk implementation is synchronous, so the guest waits on IO operations to complete on the host side.

I'll do so, but I can only make the conversion early morning or late evening, so it won't happen before tomorrow.

The reason why I'm on vmdk is that these were migrated from VMware server. The free VMware server isn't much fun anymore, and I just need "simple" virtualization, no big-iron enterprise BS compliance. ;-)

Thanks,

-martin

Signature · 2 lines
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--