KVM Test result, kernel ff5bdac.., userspace eb2fd67.. -- One New Issue

Hi All,

This is today's KVM test result against kvm.git ff5bdac4be0230e0bb33e4208ac0a91343c72929 and kvm-userspace.git eb2fd67cbecdb573f908697ed41b81ee312372bd.

One New Issue:
================================================
1. Guests crash while rebooting
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1976075&group_id=180599

Five Old Issues:
================================================
2. Fail to save restore and live migrate
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1976075&group_id=180599

3. netperf causes linux guest with virtnet kernel panic
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1972449&group_id=180599

4. XP crashes while rebooting
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1959452&group_id=180599

5.Cannot boot guests with hugetlbfs
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1941302&group_id=180599

6. failure to migrate guests with more than 4GB of RAM
https://sourceforge.net/tracker/index.php?func=detail&aid=1971512&group_id=180599&atid=893831

Test environment
================================================ Platform Woodcrest
CPU 4
Memory size 8G'

Details
================================================
IA32-pae: 1. boot guest with 256M memory PASS
2. boot guest with 1500M memory PASS
3. boot 4 same guest in parallel PASS
4. boot two windows xp guest PASS
5. boot linux and windows guest in parallel PASS
6. save/restore 32-bit HVM guests FAIL
7. save/restore 32-bit HVM guests with 4 vcpus FAIL
8. live migration 32-bit HVM guests FAIL
9. live migration 32-bit HVM guests with 4 vcpus FAIL
10. boot base kernel linux PASS
11. kernel build on SMP linux guest PASS
12. LTP on linux guest PASS
13. boot Windows 2000 without ACPI PASS
14. boot Windows 2000 with ACPI enabled PASS
15. boot Windows 2003 with ACPI enabled PASS
16. boot Windows xp with ACPI enabled PASS
17. boot Windows vista with ACPI enabled PASS
18. boot SMP Windows 2000 with ACPI enabled PASS
19. boot SMP Windows 2003 with ACPI enabled PASS
20. boot SMP Windows xp with ACPI enabled PASS
21. boot SMP Windows 2008 with ACPI enabled PASS

================================================
IA32e: 1. boot 32-bit guest with 256M memory PASS
2. boot 64-bit guest with 256M memory PASS
3. boot 32-bit guest with 1500M memory PASS
4. boot 64-bit guest with 1500M memory PASS
5. boot 4G pae guest PASS
6. boot 4G 64-bit guest PASS
7. boot four 32-bit guest in parallel PASS
8. boot four 64-bit guest in parallel PASS
9. boot two 32-bit windows xp in parallel PASS
10. boot 32-bit linux and 32 bit windows guest in parallel PASS

11. boot four 32-bit different guest in para                             PASS
12. save/restore 32-bit linux guests                                     FAIL
13. save/restore 64-bit linux guests                                     FAIL

14. save/restore 64-bit linux guests with 4 vcpus FAIL
15. save/restore 32-bit linux guests with 4 vcpus FAIL
16. live migration 64bit linux guests FAIL
17. live migration 32bit linux guests FAIL
18. live migration 64bit linux guests with 4 vcpus FAIL
19. live migration 32bit linux guests with 4 vcpus FAIL
20. boot 32-bit x-server PASS 21. kernel build in 32-bit linux guest OS PASS
22. kernel build in 64-bit linux guest OS PASS
23. LTP on 32-bit linux guest OS PASS
24. LTP on 64-bit linux guest OS PASS
25. boot 64-bit guests with ACPI enabled PASS
26. boot 32-bit Windows 2000 without ACPI PASS
27. boot 32-bit Windows xp without ACPI PASS
28. boot 64-bit Windows xp with ACPI enabled PASS
29. boot 64-bit Windows vista with ACPI enabled PASS
30. boot 32-bit SMP Windows 2000 with ACPI enabled PASS

31. boot 32-bit SMP windows 2003 with ACPI enabled          PASS
32. boot 32-bit SMP Windows xp with ACPI enabled            PASS
33. boot 64-bit SMP Windows vista with ACPI enabled         PASS
34. boot 32-bit SMP windows 2008 with ACPI enabled          PASS
35. boot 64-bit SMP windows 2003 with ACPI enabled          PASS

36. boot 64-bit SMP windows XP with ACPI enabled PASS
37. boot 64-bit SMP windows 2008 with ACPI enabled PASS
38. reboot 32bit windows xp guest FAIL
39. reboot 32bit windows xp guest FAIL

Report Summary on IA32-pae
Summary Test Report of Last Session
=====================================================================
Total Pass Fail NoResult Crash
=====================================================================

control_panel               9       5       4         0        0
Restart                     2       2       0         0        0
gtest                       17      16      1         0        0
=====================================================================
control_panel               9       5       4         0        0
:KVM_256M_guest_PAE_gPAE   1       1       0         0        0
:KVM_linux_win_PAE_gPAE    1       1       0         0        0
:KVM_two_winxp_PAE_gPAE    1       1       0         0        0
:KVM_SR_SMP_PAE_gPAE       1       0       1         0        0
:KVM_four_sguest_PAE_gPA   1       1       0         0        0
:KVM_1500M_guest_PAE_gPA   1       1       0         0        0
:KVM_LM_Continuity_PAE_g   1       0       1         0        0
:KVM_LM_SMP_PAE_gPAE       1       0       1         0        0
:KVM_SR_Continuity_PAE_g   1       0       1         0        0
Restart                     2       2       0         0        0
:GuestPAE_PAE_g32e         1       1       0         0        0
:BootTo32pae_PAE_g32e      1       1       0         0        0
gtest                       17      16      1         0        0
:boot_up_acpi_PAE_gPAE     1       1       0         0        0
:ltp_nightly_PAE_gPAE      1       1       0         0        0
:boot_up_acpi_xp_PAE_gPA   1       1       0         0        0
:reboot_xp_PAE_gPAE        1       0       1         0        0
:boot_up_vista_PAE_gPAE    1       1       0         0        0
:boot_up_acpi_win2k3_PAE   1       1       0         0        0
:boot_smp_acpi_win2k3_PA   1       1       0         0        0
:boot_smp_acpi_win2k_PAE   1       1       0         0        0
:boot_up_acpi_win2k_PAE_   1       1       0         0        0
:boot_smp_acpi_xp_PAE_gP   1       1       0         0        0
:boot_up_noacpi_win2k_PA   1       1       0         0        0
:boot_smp_vista_PAE_gPAE   1       1       0         0        0
:boot_base_kernel_PAE_gP   1       1       0         0        0
:boot_up_win2008_PAE_gPA   1       1       0         0        0
:bootx_PAE_gPAE            1       1       0         0        0
:boot_smp_win2008_PAE_gP   1       1       0         0        0
:kb_nightly_PAE_gPAE       1       1       0         0        0

=====================================================================
Total 28 23 5 0 0

Report Summary on IA32e
Summary Test Report of Last Session
=====================================================================
Total Pass Fail NoResult Crash
=====================================================================

control_panel               18      10      8         0        0
Restart                     3       3       0         0        0
gtest                       25      23      2         0        0
=====================================================================
control_panel               18      10      8         0        0
:KVM_4G_guest_64_g32e      1       1       0         0        0
:KVM_four_sguest_64_gPAE   1       1       0         0        0
:KVM_SR_SMP_64_g32e        1       0       1         0        0
:KVM_LM_SMP_64_g32e        1       0       1         0        0
:KVM_linux_win_64_gPAE     1       1       0         0        0
:KVM_LM_SMP_64_gPAE        1       0       1         0        0
:KVM_SR_Continuity_64_gP   1       0       1         0        0
:KVM_four_sguest_64_g32e   1       1       0         0        0
:KVM_four_dguest_64_gPAE   1       1       0         0        0
:KVM_SR_SMP_64_gPAE        1       0       1         0        0
:KVM_LM_Continuity_64_g3   1       0       1         0        0
:KVM_1500M_guest_64_gPAE   1       1       0         0        0
:KVM_LM_Continuity_64_gP   1       0       1         0        0
:KVM_1500M_guest_64_g32e   1       1       0         0        0
:KVM_SR_Continuity_64_g3   1       0       1         0        0
:KVM_256M_guest_64_gPAE    1       1       0         0        0
:KVM_two_winxp_64_gPAE     1       1       0         0        0
:KVM_256M_guest_64_g32e    1       1       0         0        0
Restart                     3       3       0         0        0
:GuestPAE_64_gPAE          1       1       0         0        0
:BootTo64_64_gPAE          1       1       0         0        0
:Guest64_64_gPAE           1       1       0         0        0
gtest                       25      23      2         0        0
:boot_up_acpi_64_gPAE      1       1       0         0        0
:boot_up_noacpi_xp_64_gP   1       1       0         0        0
:boot_base_kernel_64_gPA   1       1       0         0        0
:reboot_xp_64_g32e         1       0       1         0        0
:bootx_64_g32e             1       1       0         0        0
:boot_up_vista_64_g32e     1       1       0         0        0
:boot_smp_acpi_win2k3_64   1       1       0         0        0
:boot_smp_acpi_win2k_64_   1       1       0         0        0
:bootx_64_gPAE             1       1       0         0        0
:kb_nightly_64_gPAE        1       1       0         0        0
:boot_up_acpi_xp_64_g32e   1       1       0         0        0
:boot_up_noacpi_win2k_64   1       1       0         0        0
:boot_smp_acpi_xp_64_gPA   1       1       0         0        0
:boot_smp_acpi_xp_64_g32   1       1       0         0        0
:boot_smp_vista_64_gPAE    1       1       0         0        0
:boot_up_acpi_64_g32e      1       1       0         0        0
:reboot_xp_64_gPAE         1       0       1         0        0
:boot_base_kernel_64_g32   1       1       0         0        0
:boot_up_acpi_win2k3_64_   1       1       0         0        0
:kb_nightly_64_g32e        1       1       0         0        0
:ltp_nightly_64_g32e       1       1       0         0        0
:boot_up_vista_64_gPAE     1       1       0         0        0
:ltp_nightly_64_gPAE       1       1       0         0        0
:boot_smp_acpi_win2k3_64   1       1       0         0        0
:boot_up_noacpi_win2k3_6   1       1       0         0        0

=====================================================================
Total 46 36 10 0 0

Thanks
Yunfeng

This is a get_user_pages() with hugetlb-vma's bug, not KVM's problem, fixed by:

commit 5b23dbe8173c212d6a326e35347b038705603d39

Author: Adam Litke <[email protected]>
Date:   Wed Nov 14 16:59:33 2007 -0800

hugetlb: follow_hugetlb_page() for write access

When calling get_user_pages(), a write flag is passed in by the caller to indicate if write access is required on the faulted-in pages.

Currently, follow_hugetlb_page() ignores this flag and always faults pages for read-only access. This can cause data corruption because a device driver that calls get_user_pages() with write set will not expect COW faults to occur on the returned pages.

This patch passes the write flag down to follow_hugetlb_page() and makes sure hugetlb_fault() is called with the right write_access parameter.

Can you please upgrade your test hosts to 2.6.25 or newer?

Perhaps kvm-userspace should check for kernel version and disable hugetlb backed memory. Nasty.

Thanks John for help in tracking this down.

I'd say so. Just to close the loop here, attached is a trivial patch gleaned from 2.6.25 which is relative to 2.6.23.9 fc8 where I'd reproduced the issue. It dismisses the BIOS identify problem and gets the guest through a full userland init (without the earlier bandaid of mapping the first 2MB of physmem with 4KB pages).

-john

[email protected]

 include/linux/hugetlb.h |    4 ++--
 mm/hugetlb.c            |    7 ++++---
 mm/memory.c             |    2 +-
 3 files changed, 7 insertions(+), 6 deletions(-)

=================================================================

@@ -19,7 +19,7 @@ static inline int is_vm_hugetlb_page(str
 int hugetlb_sysctl_handler(struct ctl_table *, int, struct file *, void __user *, size_t *, loff_t *);
 int hugetlb_treat_movable_handler(struct ctl_table *, int, struct file *, void __user *, size_t *, loff_t *);
 int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *);
-int follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, struct vm_area_struct **, unsigned long *, int *, int);
+int follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, struct vm_area_struct **, unsigned long *, int *, int, int);
 void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
 void __unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
 int hugetlb_prefault(struct address_space *, struct vm_area_struct *);

@@ -105,7 +105,7 @@ static inline unsigned long hugetlb_tota
 	return 0;
 }
 
-#define follow_hugetlb_page(m,v,p,vs,a,b,i)	({ BUG(); 0; })
+#define follow_hugetlb_page(m,v,p,vs,a,b,i,w)	({ BUG(); 0; })
 #define follow_huge_addr(mm, addr, write)	ERR_PTR(-EINVAL)
 #define copy_hugetlb_page_range(src, dst, vma)	({ BUG(); 0; })
 #define hugetlb_prefault(mapping, vma)		({ BUG(); 0; })

=================================================================

@@ -1039,7 +1039,7 @@ int get_user_pages(struct task_struct *t
 
 		if (is_vm_hugetlb_page(vma)) {
 			i = follow_hugetlb_page(mm, vma, pages, vmas,
-						&start, &len, i);
+						&start, &len, i, write);
 			continue;
 		}
 

=================================================================

@@ -621,7 +621,8 @@ int hugetlb_fault(struct mm_struct *mm, 
 
 int follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			struct page **pages, struct vm_area_struct **vmas,
-			unsigned long *position, int *length, int i)
+			unsigned long *position, int *length, int i,
+			int write)
 {
 	unsigned long pfn_offset;
 	unsigned long vaddr = *position;

@@ -639,11 +640,11 @@ int follow_hugetlb_page(struct mm_struct
 		 */
 		pte = huge_pte_offset(mm, vaddr & HPAGE_MASK);
 
-		if (!pte || pte_none(*pte)) {
+		if (!pte || pte_none(*pte) || (write && !pte_write(*pte))) {
 			int ret;
 
 			spin_unlock(&mm->page_table_lock);
-			ret = hugetlb_fault(mm, vma, vaddr, 0);
+			ret = hugetlb_fault(mm, vma, vaddr, write);
 			spin_lock(&mm->page_table_lock);
 			if (!(ret & VM_FAULT_ERROR))
 				continue;

I tried to test it against 2.6.25-rc5.
The host hanged while booting a xp guest.

Thanks
Yunfeng

Yunfeng,

Good news is that I can't reproduce the problem with kvm.tip as of today. Both Win XP and Win 2000 Server boot happily (note: before those fixes 2000 would crash complaining about PFN list corruption) and all of their memory is mapped with 2MB pagetables.

Bad news is that I can't reproduce the problem at all anymore, so its hard to pinpoint which change fixed it (assuming the bug is actually fixed).

So, since there have been two important hugetlb fixes since ff5bdac4be0230e0bb33e4208ac0a91343c72929, it would be good if you can try a recent kvm.tip on the host.

I retested it against kvm.git d660add61a69df36372c73e68d2532190ca6430c. It works well again.

thanks
Yunfeng

That means if a distro fixed this bug, then it would still suffer. I don't think we can test for this? Otherwise I see no choice but a version check.

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

small test guest that replicates the issue, but I don't think its worth the trouble.