KVM Test result, kernel 873c05f.., userspace d102d75..
Hi All,
This is today's KVM test result against kvm.git 873c05fa7e6fea27090b1bf0f67a073eadb04782 and kvm-userspace.git d102d750f397b543fe620a3c77a7e5e42c483865.
In today's nightly testing, we meet host hang while booting multiple guests several times.
This issue could be easily reproduced.
Two Old Issues:
================================================
1. Booting four guests likely fails
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1919354&group_id=180599
2. Cannot boot guests with hugetlbfs
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1941302&group_id=180599
Test environment
================================================
Platform Woodcrest
CPU 4
Memory size 8G'
Details
================================================
IA32-pae:
1. boot guest with 256M memory PASS
2. boot two windows xp guest PASS
3. boot 4 same guest in parallel PASS
4. boot linux and windows guest in parallel PASS
5. boot guest with 1500M memory PASS
6. boot windows 2003 with ACPI enabled PASS
7. boot Windows xp with ACPI enabled PASS
8. boot Windows 2000 without ACPI PASS
9. kernel build on SMP linux guest PASS
10. LTP on linux guest PASS
11. boot base kernel linux PASS
12. save/restore 32-bit HVM guests PASS
13. live migration 32-bit HVM guests PASS
14. boot SMP Windows xp with ACPI enabled PASS
15. boot SMP Windows 2003 with ACPI enabled PASS
16. boot SMP Windows 2000 with ACPI enabled PASS
================================================
IA32e:
1. boot four 32-bit guest in
parallel PASS
2. boot four 64-bit guest in
parallel PASS
3. boot 4G 64-bit
guest PASS
4. boot 4G pae
guest PASS
5. boot 32-bit linux and 32 bit windows guest in parallel PASS
6. boot 32-bit guest with 1500M memory PASS
7. boot 64-bit guest with 1500M memory PASS
8. boot 32-bit guest with 256M memory PASS
9. boot 64-bit guest with 256M memory PASS
10. boot two 32-bit windows xp in parallel PASS
11. boot four 32-bit different guest in para
PASS
12. save/restore 64-bit linux guests
PASS
13. save/restore 32-bit linux guests
PASS
14. boot 32-bit SMP windows 2003 with ACPI enabled PASS
15. boot 32-bit SMP Windows 2000 with ACPI enabled PASS
16. boot 32-bit SMP Windows xp with ACPI enabled PASS
17. boot 32-bit Windows 2000 without ACPI PASS
18. boot 64-bit Windows xp with ACPI enabled PASS
19. boot 32-bit Windows xp without ACPI PASS
20. boot 64-bit UP
vista PASS
21. boot 64-bit SMP
vista PASS
22. kernel build in 32-bit linux guest OS PASS
23. kernel build in 64-bit linux guest OS PASS
24. LTP on 32-bit linux guest OS PASS
25. LTP on 64-bit linux guest OS PASS
26. boot 64-bit guests with ACPI enabled PASS
27. boot 32-bit
x-server PASS
28. boot 64-bit SMP windows XP with ACPI enabled PASS
29. boot 64-bit SMP windows 2003 with ACPI enabled PASS
30. live migration 64bit linux
guests PASS
31. live migration 32bit linux
guests PASS
32. reboot 32bit windows xp guest PASS
33. reboot 32bit windows xp guest PASS
Report Summary on IA32-pae
Summary Test Report of Last Session
=====================================================================
Total Pass Fail NoResult Crash
=====================================================================
control_panel 7 7 0 0 0 Restart 2 2 0 0 0 gtest 15 15 0 0 0 ===================================================================== control_panel 7 7 0 0 0 :KVM_LM_PAE_gPAE 1 1 0 0 0 :KVM_four_sguest_PAE_gPA 1 1 0 0 0 :KVM_256M_guest_PAE_gPAE 1 1 0 0 0 :KVM_linux_win_PAE_gPAE 1 1 0 0 0 :KVM_1500M_guest_PAE_gPA 1 1 0 0 0 :KVM_SR_PAE_gPAE 1 1 0 0 0 :KVM_two_winxp_PAE_gPAE 1 1 0 0 0 Restart 2 2 0 0 0 :GuestPAE_PAE_gPAE 1 1 0 0 0 :BootTo32pae_PAE_gPAE 1 1 0 0 0 gtest 15 15 0 0 0 :ltp_nightly_PAE_gPAE 1 1 0 0 0 :boot_up_acpi_PAE_gPAE 1 1 0 0 0 :reboot_xp_PAE_gPAE 1 1 0 0 0 :boot_up_vista_PAE_gPAE 1 1 0 0 0 :boot_up_acpi_xp_PAE_gPA 1 1 0 0 0 :boot_up_acpi_win2k3_PAE 1 1 0 0 0 :boot_base_kernel_PAE_gP 1 1 0 0 0 :boot_smp_acpi_win2k3_PA 1 1 0 0 0 :boot_smp_acpi_win2k_PAE 1 1 0 0 0 :boot_up_acpi_win2k_PAE_ 1 1 0 0 0 :boot_smp_acpi_xp_PAE_gP 1 1 0 0 0 :boot_up_noacpi_win2k_PA 1 1 0 0 0 :boot_smp_vista_PAE_gPAE 1 1 0 0 0 :bootx_PAE_gPAE 1 1 0 0 0 :kb_nightly_PAE_gPAE 1 1 0 0 0
=====================================================================
Total 24 24 0 0 0
Report Summary on IA32e
Summary Test Report of Last Session
=====================================================================
Total Pass Fail NoResult Crash
=====================================================================
control_panel 15 14 1 0 0 Restart 3 3 0 0 0 gtest 25 25 0 0 0 ===================================================================== control_panel 15 14 1 0 0 :KVM_LM_64_g64 1 1 0 0 0 :KVM_four_sguest_64_gPAE 1 1 0 0 0 :KVM_4G_guest_64_g64 1 1 0 0 0 :KVM_four_sguest_64_g64 1 1 0 0 0 :KVM_linux_win_64_gPAE 1 1 0 0 0 :KVM_1500M_guest_64_gPAE 1 1 0 0 0 :KVM_SR_64_g64 1 0 1 0 0 :KVM_LM_64_gPAE 1 1 0 0 0 :KVM_256M_guest_64_g64 1 1 0 0 0 :KVM_1500M_guest_64_g64 1 1 0 0 0 :KVM_4G_guest_64_gPAE 1 1 0 0 0 :KVM_SR_64_gPAE 1 1 0 0 0 :KVM_256M_guest_64_gPAE 1 1 0 0 0 :KVM_two_winxp_64_gPAE 1 1 0 0 0 :KVM_four_dguest_64_gPAE 1 1 0 0 0 Restart 3 3 0 0 0 :GuestPAE_64_gPAE 1 1 0 0 0 :BootTo64_64_gPAE 1 1 0 0 0 :Guest64_64_gPAE 1 1 0 0 0 gtest 25 25 0 0 0 :boot_up_acpi_64_gPAE 1 1 0 0 0 :boot_up_noacpi_xp_64_gP 1 1 0 0 0 :boot_smp_acpi_xp_64_g64 1 1 0 0 0 :boot_base_kernel_64_gPA 1 1 0 0 0 :boot_smp_acpi_win2k3_64 1 1 0 0 0 :boot_smp_acpi_win2k_64_ 1 1 0 0 0 :boot_base_kernel_64_g64 1 1 0 0 0 :bootx_64_gPAE 1 1 0 0 0 :kb_nightly_64_gPAE 1 1 0 0 0 :ltp_nightly_64_g64 1 1 0 0 0 :boot_up_acpi_64_g64 1 1 0 0 0 :boot_up_noacpi_win2k_64 1 1 0 0 0 :boot_smp_acpi_xp_64_gPA 1 1 0 0 0 :boot_smp_vista_64_gPAE 1 1 0 0 0 :boot_up_acpi_win2k3_64_ 1 1 0 0 0 :reboot_xp_64_gPAE 1 1 0 0 0 :bootx_64_g64 1 1 0 0 0 :boot_up_vista_64_g64 1 1 0 0 0 :boot_smp_vista_64_g64 1 1 0 0 0 :boot_up_acpi_xp_64_g64 1 1 0 0 0 :boot_up_vista_64_gPAE 1 1 0 0 0 :ltp_nightly_64_gPAE 1 1 0 0 0 :boot_smp_acpi_win2k3_64 1 1 0 0 0 :boot_up_noacpi_win2k3_6 1 1 0 0 0 :kb_nightly_64_g64 1 1 0 0 0
=====================================================================
Total 43 42 1 0 0
Best Regards,
Yunfeng
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 7 lines · Yunfeng Zhao
Hi All,
This is today's KVM test result against kvm.git
873c05fa7e6fea27090b1bf0f67a073eadb04782and kvm-userspace.gitd102d750f397b543fe620a3c77a7e5e42c483865.
I suspect 873c05fa7e6fea27090b1bf0f67a073eadb04782 itself, it's the only thing that has any chance of badness.
Marcelo, any idea? Perhaps due to load, interrupts accumulate and can't be injected fast enough?
These tests are run on a 2.6.22 host, which has a hacked smp_call_function_single() in external-module-compat.h, which may exaberate the problem.
Signature · 8 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 10 lines · On Thursday 24 April 2008 19:37:03 Avi Kivity wrote:
Quoted 5 lines · Yunfeng Zhao
Hi All,
This is today's KVM test result against kvm.git
873c05fa7e6fea27090b1bf0f67a073eadb04782and kvm-userspace.gitd102d750f397b543fe620a3c77a7e5e42c483865.I suspect
873c05fa7e6fea27090b1bf0f67a073eadb04782itself, it's the only thing that has any chance of badness.
Quoted 6 lines
Marcelo, any idea? Perhaps due to load, interrupts accumulate and can't be injected fast enough?
These tests are run on a 2.6.22 host, which has a hacked
smp_call_function_single()inexternal-module-compat.h, which may exaberate the problem.
Yeah, I suspect the commit too(I tried tip without that, and found mostly alright). In fact, I didn't use kvm_vcpu_kick() just because that I found this function may causing hang on my host... But I didn't do more investigate so I can't tell what's wrong, then I just chose way to keep it working... I am sorry for not clarify...
--
Thanks
Yang, Sheng
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 21 lines · Sheng
Quoted 11 lines · On Thursday 24 April 2008 19:37:03 Avi Kivity wrote:
Quoted 6 lines · Yunfeng Zhao
Hi All,
This is today's KVM test result against kvm.git
873c05fa7e6fea27090b1bf0f67a073eadb04782and kvm-userspace.gitd102d750f397b543fe620a3c77a7e5e42c483865.I suspect
873c05fa7e6fea27090b1bf0f67a073eadb04782itself, it's the only thing that has any chance of badness.Quoted 7 lines
Marcelo, any idea? Perhaps due to load, interrupts accumulate and can't be injected fast enough?
These tests are run on a 2.6.22 host, which has a hacked
smp_call_function_single()inexternal-module-compat.h, which may exaberate the problem.
Quoted 6 lines
Yeah, I suspect the commit
too(I tried tip without that, and found mostly alright). In fact, I didn't usekvm_vcpu_kick()just because that I found this function may causing hang on my host... But I didn't do more investigate so I can't tell what's wrong, then I just chose way to keep it working... I am sorry for not clarify...
I think smp_call_function_single() is miscompiled when using the compatibility code. I took it out-of-line to be sure (it is now in kernel/external-module-compat.c).
No evidence, but...
Signature · 8 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 31 lines · Avi Kivity
Quoted 23 lines · Sheng
Quoted 12 lines · On Thursday 24 April 2008 19:37:03 Avi Kivity wrote:
Quoted 6 lines · Yunfeng Zhao
Hi All,
This is today's KVM test result against kvm.git
873c05fa7e6fea27090b1bf0f67a073eadb04782and kvm-userspace.gitd102d750f397b543fe620a3c77a7e5e42c483865.I suspect
873c05fa7e6fea27090b1bf0f67a073eadb04782itself, it's the only
thing that has any chance of badness.Quoted 8 lines
Marcelo, any idea? Perhaps due to load, interrupts accumulate and can't
be injected fast enough?These tests are run on a 2.6.22 host, which has a hacked
smp_call_function_single()inexternal-module-compat.h, which may exaberate the problem.Quoted 6 lines
Yeah, I suspect the commit
too(I tried tip without that, and found mostly alright). In fact, I didn't usekvm_vcpu_kick()just because that I found this function may causing hang on my host... But I didn't do more investigate so I can't tell what's wrong, then I just chose way to keep it working... I am sorry for not clarify...
Quoted 6 lines
I think
smp_call_function_single()is miscompiled when using the compatibility code. I took it out-of-line to be sure (it is now inkernel/external-module-compat.c).No evidence, but...
Ok. __pit_timer_fn() is called from an interrupt, which then calls smp_call_function_single(), which calls spin_lock(). If we've already taken the lock, we hang.
Signature · 8 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 5 lines · Avi Kivity
Ok.
__pit_timer_fn()is called from an interrupt, which then callssmp_call_function_single(), which callsspin_lock(). If we've already taken the lock, we hang.
Ah. Just adding a "me too"; I didn't get a chance to debug it yesterday, but I was seeing similar problems. If I disabled in-kernel pit with -no-kvm-pit, all was well.
Chris Lalancette
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 8 lines · Chris Lalancette
Quoted 5 lines · Avi Kivity
Ok.
__pit_timer_fn()is called from an interrupt, which then callssmp_call_function_single(), which callsspin_lock(). If we've already taken the lock, we hang.
Quoted 4 lines
Ah. Just adding a "me too"; I didn't get a chance to debug it yesterday, but I was seeing similar problems. If I disabled in-kernel pit with -no-kvm-pit, all was well.
How to fix it, though? the only idea that comes to mind is to affine the hrtimer with vcpu0 (like the local apic timers) which would mean we only need to unwait the waitqueue, and never need to send the IPI. Would slightly improve performance as well.
Signature · 8 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 16 lines · Avi Kivity
Quoted 8 lines · Chris Lalancette
Quoted 5 lines · Avi Kivity
Ok.
__pit_timer_fn()is called from an interrupt, which then callssmp_call_function_single(), which callsspin_lock(). If we've already taken the lock, we hang.Quoted 6 lines
Ah. Just adding a "me too"; I didn't get a chance to debug it yesterday, but I
was seeing similar problems. If I disabled in-kernel pit with -no-kvm-pit, all
was well.
Quoted 4 lines
How to fix it, though? the only idea that comes to mind is to affine the hrtimer with vcpu0 (like the local apic timers) which would mean we only need to unwait the waitqueue, and never need to send the IPI. Would slightly improve performance as well.
Yes, agree.
For now I think just revert
arch/x86/kvm/i8254.c +2 -4
@@ -200,10 +200,8 @@ int __pit_timer_fn(struct kvm_kpit_state *ps)atomic_inc (&pt->pending);smp_mb__after_atomic_inc (); -if (vcpu0 &&waitqueue_active (&vcpu0->wq)) { - vcpu0->arch.mp_state =KVM_MP_STATE_RUNNABLE ; -wake_up_interruptible (&vcpu0->wq); - } +if (vcpu0) +kvm_vcpu_kick (vcpu0);
And add a big fat FIXME.
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Quoted 26 lines · Marcelo Tosatti
Quoted 18 lines · Avi Kivity
Quoted 9 lines · Chris Lalancette
Quoted 6 lines · Avi Kivity
Ok.
__pit_timer_fn()is called from an interrupt, which then callssmp_call_function_single(), which callsspin_lock(). If we've already taken the lock, we hang.Quoted 7 lines
Ah. Just adding a "me too"; I didn't get a chance to debug it yesterday, but I
was seeing similar problems. If I disabled in-kernel pit with -no-kvm-pit, all
was well.Quoted 5 lines
How to fix it, though? the only idea that comes to mind is to affine the hrtimer with vcpu0 (like the local apic timers) which would mean we only need to unwait the waitqueue, and never need to send the IPI. Would slightly improve performance as well.
Quoted 4 lines
Yes, agree.
For now I think just revert
I committed this, so this should be fixed for now. I'm not sure hrtimer migration would work 100% reliably (suppose it fired just after a vcpu migration) so I think a queue_work is better.
Signature · 8 lines
error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone