Skip to contentSkip to conversation

2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)

DISCUSSION Nikola Ciprich 4 participants 9 messages May 8, 2011 – May 9, 2011 KVM

lore
Share & tools
lore

Nikola Ciprich

author
#0 lore raw

Hello everyboy,
while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... Either they don't finish booting at all, or boot but hang soon after..
Such a hanged guest fully loads all host cpus..
The host is 6core x86_64 runnig 2.6.37.6 with 24GB RAM.

kvm_stat:

 kvm_exit(EXTERNAL_INTERRUPT)                 15682    1643
 kvm_exit                                     15514    1643
 kvm_entry                                    15416    1643
 kvm_set_irq                                      1	  0
 kvm_msi_set_irq                                  1	  0
 kvm_apic_accept_irq                              1	  0
 kvm_exit(VMCLEAR)                                6       0
 kvm_exit(VMON)                                   6       0
 kvm_exit(PAUSE_INSTRUCTION)                      5       0
 kvm_exit(MCE_DURING_VMENTRY)                     5       0
 kvm_exit(MWAIT_INSTRUCTION)                      5       0
 kvm_exit(DR_ACCESS)                              5       0
 kvm_exit(EPT_VIOLATION)                          5       0
 kvm_exit(NMI_WINDOW)                             5       0
 kvm_exit(VMPTRLD)                                5       0
 kvm_exit(TASK_SWITCH)                            5       0
 kvm_exit(VMREAD)                                 5       0
 kvm_exit(VMLAUNCH)                               5       0
 kvm_exit(RDPMC)                                  5       0

perf top:

               16.00 10.9% add_preempt_count        [kernel.kallsyms]                                        
               16.00 10.9% do_raw_spin_lock         [kernel.kallsyms]                                        
               15.00 10.2% sub_preempt_count        [kernel.kallsyms]                                        
                8.00  5.4% irq_exit                 [kernel.kallsyms]                                        
                7.00  4.8% vmx_vcpu_run             /lib/modules/2.6.37lb.09/kernel/arch/x86/kvm/kvm-intel.ko
                7.00  4.8% page_fault               [kernel.kallsyms]                                        
                5.00  3.4% mempool_free             [kernel.kallsyms]

info registers:
RAX=0000000000f42400 RBX=ffffffff81533f00 RCX=0000000000000016 RDX=00077358f5000000 RSI=000000001dcd6500 RDI=0000000000000001 RBP=ffff880009a03ee8 RSP=ffff880009a03ee8 R8 =0000000000000016 R9 =000000000000000a R10=00000000ffffffff R11=0000000000000000 R12=2a4d17d38f3303c1 R13=ffffffff815fd000 R14=ffffffff81592140 R15=0000000000093510 RIP=ffffffff810767cb RFL=00000006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 000fffff 00000000
GS =0000 ffff880009a00000 000fffff 00000000
LDT=0000 0000000000000000 000fffff 00000000
TR =0040 ffff880009a11880 00002087 00008b00 DPL=0 TSS64-busy
GDT= ffff880009a04000 0000007f
IDT= ffffffff815fd000 00000fff
CR0=8005003b CR2=00007f424e540700 CR3=000000021690c000 CR4=000006f0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=0000000000000000000000003c23d70a XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000 XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000 XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000 XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000 XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000

info cpus:
* CPU #0: pc=0xffffffff8105d4a0 thread_id=19639 CPU #1: pc=0xffffffff81013140 thread_id=19640 CPU #2: pc=0xffffffff8102a1b6 (halted) thread_id=19641 CPU #3: pc=0xffffffff81341521 thread_id=19642 CPU #4: pc=0xffffffff810415d8 thread_id=19643 CPU #5: pc=0xffffffff811ca521 thread_id=19644 CPU #6: pc=0xffffffff81013140 thread_id=19646 CPU #7: pc=0xffffffff8102a1b6 (halted) thread_id=19647

and here are trace-cmds for all cpus:
http://nik.lbox.cz/public/trace-cmd.tar.bz2

Could somebody please have a look at this?

I also tried 2.6.38.5, but the result is the same...

cheers
nik

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#1 lore raw
Quoted 2 lines · Nikola Ciprich

Hello everyboy,
while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled...

There were about 10 bugfixes pushed to 2.6.32.y stable series, some of them were for kvm-clock, and some were for problems which manifested itself like you described. You may actually take a look which guests you're booting. FWIW.

/mjt

author
#2 lore raw

OK,
I see.. the problem is, that I'm trying to hunt down bug causing hangs
when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock..
So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful..
cheers
n.

Quoted 15 lines · Michael Tokarev
Quoted 2 lines · Nikola Ciprich

Hello everyboy,
while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled...

There were about 10 bugfixes pushed to 2.6.32.y stable series, some of them were for kvm-clock, and some were for problems which manifested itself like you described. You may actually take a look which guests you're booting. FWIW.

/mjt
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#3 lore raw
Quoted 6 lines · Nikola Ciprich

OK,
I see.. the problem is, that I'm trying to hunt down bug causing hangs
when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock..
So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful..
cheers

Have you tried enabling gdbserver in the qemu monitor and then attaching gdb to the guest once it hangs?

David

#4 lore raw
In reply to Nikola Ciprich
Quoted 8 lines · Nikola Ciprich

OK,
I see.. the problem is, that I'm trying to hunt down bug causing hangs
when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock..
So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful..
cheers
n.

What are you bisecting, the host kernel or the guest kernel, and what version is the host kernel?

author
#5 lore raw

The guest, because latest kernels do not suffer this problem, so I'd like to find fix so it can be pushed to -stable (we're using 2.6.32.x) host is currently 2.6.37 (and i'm currently testing 2.6.38 as well) n.

Quoted 13 lines · Zachary Amsden
Quoted 8 lines · Nikola Ciprich

OK,
I see.. the problem is, that I'm trying to hunt down bug causing hangs
when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock..
So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful..
cheers
n.

What are you bisecting, the host kernel or the guest kernel, and what version is the host kernel?

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#6 lore raw
Quoted 4 lines · Nikola Ciprich

The guest, because latest kernels do not suffer this problem, so I'd like to find fix so it can be pushed to -stable (we're using 2.6.32.x) host is currently 2.6.37 (and i'm currently testing 2.6.38 as well) n.

That's a pretty wide range to be bisecting, and I think we know for a fact there were some kvmclock related bugs in that range.

If you are looking for something causing problems with tcpdump, I'd suggest getting rid of kvmclock in your testing and using TSC instead; if you're looking to verify that kvmclock related changed have been backported to -stable, rather than bisect and run into bugs, it would probably be faster to check the commit logs for arch/x86/kvm/x86.c and make sure you're not missing anything from me or Glauber that has been applied to the most recent branch.

Zach

author
#7 lore raw
Quoted 2 lines

That's a pretty wide range to be bisecting, and I think we know for a fact there were some kvmclock related bugs in that range.

thats true, I might try to pick those that seem related and see if it helpts..

Quoted 3 lines

If you are looking for something causing problems with tcpdump, I'd suggest getting rid of kvmclock in your testing and using TSC instead;

that's the problem, I can't reproduce the problems without kvm-clock enabled, so it must be related to it somehow..

Quoted 5 lines

if you're looking to verify that kvmclock related changed have been backported to -stable, rather than bisect and run into bugs, it would probably be faster to check the commit logs for arch/x86/kvm/x86.c and make sure you're not missing anything from me or Glauber that has been applied to the most recent branch.

yup, I'll try and report...
thanks for the hints!
n.

Quoted 3 lines

Zach

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
author
#8 lore raw
In reply to Nikola Ciprich

(CC Zachary)
well, I should also note that while testing 2.6.37 host, I had Zach's patch fixing guest clock regression applied...
n.

Quoted 110 lines · Nikola Ciprich

Hello everyboy,
while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... Either they don't finish booting at all, or boot but hang soon after..
Such a hanged guest fully loads all host cpus..
The host is 6core x86_64 runnig 2.6.37.6 with 24GB RAM.

kvm_stat:

 kvm_exit(EXTERNAL_INTERRUPT)                 15682    1643
 kvm_exit                                     15514    1643
 kvm_entry                                    15416    1643
 kvm_set_irq                                      1	  0
 kvm_msi_set_irq                                  1	  0
 kvm_apic_accept_irq                              1	  0
 kvm_exit(VMCLEAR)                                6       0
 kvm_exit(VMON)                                   6       0
 kvm_exit(PAUSE_INSTRUCTION)                      5       0
 kvm_exit(MCE_DURING_VMENTRY)                     5       0
 kvm_exit(MWAIT_INSTRUCTION)                      5       0
 kvm_exit(DR_ACCESS)                              5       0
 kvm_exit(EPT_VIOLATION)                          5       0
 kvm_exit(NMI_WINDOW)                             5       0
 kvm_exit(VMPTRLD)                                5       0
 kvm_exit(TASK_SWITCH)                            5       0
 kvm_exit(VMREAD)                                 5       0
 kvm_exit(VMLAUNCH)                               5       0
 kvm_exit(RDPMC)                                  5       0

perf top:

               16.00 10.9% add_preempt_count        [kernel.kallsyms]                                        
               16.00 10.9% do_raw_spin_lock         [kernel.kallsyms]                                        
               15.00 10.2% sub_preempt_count        [kernel.kallsyms]                                        
                8.00  5.4% irq_exit                 [kernel.kallsyms]                                        
                7.00  4.8% vmx_vcpu_run             /lib/modules/2.6.37lb.09/kernel/arch/x86/kvm/kvm-intel.ko
                7.00  4.8% page_fault               [kernel.kallsyms]                                        
                5.00  3.4% mempool_free             [kernel.kallsyms]

info registers:
RAX=0000000000f42400 RBX=ffffffff81533f00 RCX=0000000000000016 RDX=00077358f5000000 RSI=000000001dcd6500 RDI=0000000000000001 RBP=ffff880009a03ee8 RSP=ffff880009a03ee8 R8 =0000000000000016 R9 =000000000000000a R10=00000000ffffffff R11=0000000000000000 R12=2a4d17d38f3303c1 R13=ffffffff815fd000 R14=ffffffff81592140 R15=0000000000093510 RIP=ffffffff810767cb RFL=00000006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 000fffff 00000000
GS =0000 ffff880009a00000 000fffff 00000000
LDT=0000 0000000000000000 000fffff 00000000
TR =0040 ffff880009a11880 00002087 00008b00 DPL=0 TSS64-busy
GDT= ffff880009a04000 0000007f
IDT= ffffffff815fd000 00000fff
CR0=8005003b CR2=00007f424e540700 CR3=000000021690c000 CR4=000006f0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=0000000000000000000000003c23d70a XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000 XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000 XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000 XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000 XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000

info cpus:
* CPU #0: pc=0xffffffff8105d4a0 thread_id=19639 CPU #1: pc=0xffffffff81013140 thread_id=19640 CPU #2: pc=0xffffffff8102a1b6 (halted) thread_id=19641 CPU #3: pc=0xffffffff81341521 thread_id=19642 CPU #4: pc=0xffffffff810415d8 thread_id=19643 CPU #5: pc=0xffffffff811ca521 thread_id=19644 CPU #6: pc=0xffffffff81013140 thread_id=19646 CPU #7: pc=0xffffffff8102a1b6 (halted) thread_id=19647

and here are trace-cmds for all cpus:
http://nik.lbox.cz/public/trace-cmd.tar.bz2

Could somebody please have a look at this?

I also tried 2.6.38.5, but the result is the same...

cheers
nik

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------