just a dump
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore
vm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
dmesg:
device tap0 entered promiscuous mode
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap1: no IPv6 routers present
tap0: no IPv6 routers present
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
BUG: unable to handle kernel NULL pointer dereference at (null)
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap1: no IPv6 routers present
tap0: no IPv6 routers present
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60 PGD 1a0c4b067 PUD 1a0c4a067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource CPU 0 Modules linked in: Pid: 2945, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System Product Name RIP: 0010:[<ffffffff80216ee2>] [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60 RSP: 0018:ffff8801a0f979d8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 00000000000e1000 RSI: af7dc78941006c5f RDI: ffff8801a0d90000 RBP: ffff8801a0f979e8 R08: 0000000000000006 R09: 0000000000000006 R10: 0000000000000000 R11: 0000000000000000 R12: af7dc78941006c5f R13: ffff8801908b1180 R14: ffff88019e778c60 R15: ffff8801a0d90000 FS: 0000000041591950(0063) GS:ffff880028034000(0000) knlGS:fffff800017ce500 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001a3d7c000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process qemu-system-x86 (pid: 2945, threadinfo ffff8801a0f96000, task ffff8801afb12d00) Stack: 0000000000000010 0000000000000000 ffff8801a0f97a28 ffffffff80216fce ffff8801a0f97a28 af7dc78941006c5f 0000000000000000 0000000000000000 0000000000000180 ffff88019e778c60 ffff8801a0f97ac8 ffffffff8021ad8d Call Trace: [<ffffffff80216fce>] rmap_remove+0xae/0x200 [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30 [<ffffffff80216edb>] ? gfn_to_rmap+0x1b/0x60 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13 [<ffffffff8022b9ce>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0 [<ffffffff8027d154>] ? sys_futex+0x84/0x130 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0 [<ffffffff802cc08b>] ? sys_lseek+0x7b/0x80 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b Code: 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 89 d3 4c 89 64 24 08 49 89 f4 e8 a5 36 ff ff 85 db 48 89 c1 75 1e <4c> 2b 20 4a 8d 14 e5 00 00 00 00 48 03 50 18 48 8b 1c 24 4c 8b RIP [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60 RSP <ffff8801a0f979d8> CR2: 0000000000000000 ---[ end trace 5b490dc3e478a040 ]---
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
CE: hpet increasing min_delta_ns to 15000 nsec
one vm was still running.
Signature · 1 line
Hans
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1
Quoted 3 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
device tap0 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap0: no IPv6 routers present
tap1: no IPv6 routers present
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0rmap_remove: ffff880100de5500 8 0->BUG
------------[ cut here ]------------ kernel BUG at arch/x86/kvm/mmu.c:576! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource CPU 1 Modules linked in: Pid: 2925, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System Product Name RIP: 0010:[<ffffffff80217071>] [<ffffffff80217071>] rmap_remove+0x151/0x200 RSP: 0018:ffff8801a0d379f8 EFLAGS: 00010292 RAX: 000000000000002a RBX: 0000000000000008 RCX: ffffffff809a3b40 RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a34 RBP: ffff8801a0d37a28 R08: 0000000000008777 R09: 00000000ffffffff R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff880100de5500 R14: ffff880101e23580 R15: ffff8801a0e1c000 FS: 000000004270d950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffaa000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000014a8c18 CR3: 00000001a0c62000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process qemu-system-x86 (pid: 2925, threadinfo ffff8801a0d36000, task ffff8801af3605a0) Stack: ffff8801a0d37a28 0000000000000000 0000000000000000 0000000000000000 0000000000000500 ffff880101e23580 ffff8801a0d37ac8 ffffffff8021ad8d 0000000000000000 ffff880100000000 000000000003020d 000000000016e772 Call Trace: [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120 [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30 [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7 eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 b8 2e 7f 80 e8 1f 29 04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84 RIP [<ffffffff80217071>] rmap_remove+0x151/0x200 RSP <ffff8801a0d379f8> ---[ end trace c11385df745a1fea ]--- BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 PGD 1a0ca8067 PUD 1a0ca9067 PMD 0 Oops: 0000 [#2] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource CPU 0 Modules linked in: Pid: 2926, comm: qemu-system-x86 Tainted: G D 2.6.30-rc5 #3 System Product Name RIP: 0010:[<ffffffff80216b4c>] [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 RSP: 0018:ffff8801a0da57a8 EFLAGS: 00010292 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002b RDX: ffffe20000000000 RSI: ffff8800ccac0220 RDI: 0000000000000000 RBP: ffff8801a0da57b8 R08: 000000000000006a R09: ffff8800ccd85e70 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800ccac0220 R13: ffff8800ccd85dc0 R14: 0000000000000044 R15: ffff8801a0db0000 FS: 0000000040fbc950(0063) GS:ffff880028034000(0000) knlGS:000007fffffd5000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 00000001a0c63000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process qemu-system-x86 (pid: 2926, threadinfo ffff8801a0da4000, task ffff8801ae971c20) Stack: ffff8800ccd85590 000000000000007a ffff8801a0da5948 ffffffff80217323 ffff8801a0da5808 0000000000000056 ffff8800ccd85dc0 ffffe20000000000 ffff8801030f8160 0000000000000003 ffff880103f87000 ffffffff000001b8 Call Trace: [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0 [<ffffffff8020a207>] ? mark_page_dirty+0x27/0x60 [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30 [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60 [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70 [<ffffffff8021993c>] kvm_mmu_pte_write+0x72c/0x910 [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320 [<ffffffff8020a31c>] ? kvm_write_guest_page+0x7c/0x80 [<ffffffff8020fb0d>] emulator_write_phys+0x4d/0x70 [<ffffffff80211785>] emulator_write_emulated_onepage+0x95/0x120 [<ffffffff80211880>] emulator_write_emulated+0x70/0x90 [<ffffffff8021d11e>] x86_emulate_insn+0x3de/0x4d30 [<ffffffff8021bcef>] ? decode_register_operand+0x8f/0x100 [<ffffffff8021c50c>] ? x86_decode_insn+0x6dc/0xf10 [<ffffffff8020e710>] ? kvm_find_cpuid_entry+0xf0/0x110 [<ffffffff8020f91f>] emulate_instruction+0x15f/0x2f0 [<ffffffff802191da>] kvm_mmu_page_fault+0x5a/0x90 [<ffffffff80226e7f>] pf_interception+0x7f/0x190 [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70 [<ffffffff8022822a>] handle_exit+0x14a/0x2c0 [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [<ffffffff8026e680>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [<ffffffff803ea171>] ? security_file_permission+0x11/0x20 [<ffffffff802cba0f>] ? do_readv_writev+0x14f/0x1d0 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0 [<ffffffff802cc4c1>] ? sys_writev+0x81/0x90 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f RIP [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 RSP <ffff8801a0da57a8> CR2: 0000000000000058 ---[ end trace c11385df745a1feb ]---
Starting with a clear disk cach makes the change of this happening bigger. The first time the screen mode of one of the vm's just changed to the left to right walking progress bar. This time I brought one of the windows to the front late in de bootprocess (the windows applying computer settings screen).
Signature · 1 line
Hans
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1
Quoted 5 lines
Quoted 4 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
Hans,
The oopses below point to the possibility of a hardware problem, similar to:
https://bugzilla.redhat.com/show_bug.cgi?id=480779
Can you please rule it out with memtest86?
Quoted 152 lines
device tap0 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap0: no IPv6 routers present
tap1: no IPv6 routers present
kvm: 2915: cpu0 unimplemented perfctr wrmsr:0xc0010000data0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr:0xc0010001data0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr:0xc0010002data0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr:0xc0010003data0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr:0xc0010000data0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr:0xc0010001data0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr:0xc0010002data0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr:0xc0010003data0x0rmap_remove:ffff880100de55008 0->BUG------------[ cut here ]------------ kernel BUG at arch/x86/kvm/mmu.c:576! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource CPU 1 Modules linked in: Pid: 2925, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System Product Name RIP: 0010:[<ffffffff80217071>] [<ffffffff80217071>] rmap_remove+0x151/0x200 RSP: 0018:ffff8801a0d379f8 EFLAGS: 00010292 RAX: 000000000000002a RBX: 0000000000000008 RCX: ffffffff809a3b40 RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a34 RBP: ffff8801a0d37a28 R08: 0000000000008777 R09: 00000000ffffffff R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff880100de5500 R14: ffff880101e23580 R15: ffff8801a0e1c000 FS: 000000004270d950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffaa000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000014a8c18 CR3: 00000001a0c62000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process qemu-system-x86 (pid: 2925, threadinfo ffff8801a0d36000, task ffff8801af3605a0) Stack: ffff8801a0d37a28 0000000000000000 0000000000000000 0000000000000000 0000000000000500 ffff880101e23580 ffff8801a0d37ac8 ffffffff8021ad8d 0000000000000000 ffff880100000000 000000000003020d 000000000016e772 Call Trace: [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120 [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30 [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 b8 2e 7f 80 e8 1f 29
04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84 RIP [<ffffffff80217071>]rmap_remove+0x151/0x200
RSP <ffff8801a0d379f8>---[ end trace c11385df745a1fea ]--- BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 PGD 1a0ca8067 PUD 1a0ca9067 PMD 0 Oops: 0000 [#2] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource CPU 0 Modules linked in: Pid: 2926, comm: qemu-system-x86 Tainted: G D 2.6.30-rc5 #3 System Product Name RIP: 0010:[<ffffffff80216b4c>] [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 RSP: 0018:ffff8801a0da57a8 EFLAGS: 00010292 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002b RDX: ffffe20000000000 RSI: ffff8800ccac0220 RDI: 0000000000000000 RBP: ffff8801a0da57b8 R08: 000000000000006a R09: ffff8800ccd85e70 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800ccac0220 R13: ffff8800ccd85dc0 R14: 0000000000000044 R15: ffff8801a0db0000 FS: 0000000040fbc950(0063) GS:ffff880028034000(0000) knlGS:000007fffffd5000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 00000001a0c63000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process qemu-system-x86 (pid: 2926, threadinfo ffff8801a0da4000, task ffff8801ae971c20) Stack: ffff8800ccd85590 000000000000007a ffff8801a0da5948 ffffffff80217323 ffff8801a0da5808 0000000000000056 ffff8800ccd85dc0 ffffe20000000000 ffff8801030f8160 0000000000000003 ffff880103f87000 ffffffff000001b8 Call Trace: [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0 [<ffffffff8020a207>] ? mark_page_dirty+0x27/0x60 [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30 [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60 [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70 [<ffffffff8021993c>] kvm_mmu_pte_write+0x72c/0x910 [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320 [<ffffffff8020a31c>] ? kvm_write_guest_page+0x7c/0x80 [<ffffffff8020fb0d>] emulator_write_phys+0x4d/0x70 [<ffffffff80211785>] emulator_write_emulated_onepage+0x95/0x120 [<ffffffff80211880>] emulator_write_emulated+0x70/0x90 [<ffffffff8021d11e>] x86_emulate_insn+0x3de/0x4d30 [<ffffffff8021bcef>] ? decode_register_operand+0x8f/0x100 [<ffffffff8021c50c>] ? x86_decode_insn+0x6dc/0xf10 [<ffffffff8020e710>] ? kvm_find_cpuid_entry+0xf0/0x110 [<ffffffff8020f91f>] emulate_instruction+0x15f/0x2f0 [<ffffffff802191da>] kvm_mmu_page_fault+0x5a/0x90 [<ffffffff80226e7f>] pf_interception+0x7f/0x190 [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70 [<ffffffff8022822a>] handle_exit+0x14a/0x2c0 [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [<ffffffff8026e680>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [<ffffffff803ea171>] ? security_file_permission+0x11/0x20 [<ffffffff802cba0f>] ? do_readv_writev+0x14f/0x1d0 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0 [<ffffffff802cc4c1>] ? sys_writev+0x81/0x90 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 662e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f RIP [<
ffffffff80216b4c>]mmu_page_remove_parent_pte+0xc/0x100
RSP <ffff8801a0da57a8>
CR2: 0000000000000058---[ end trace c11385df745a1feb ]---Starting with a clear disk cach makes the change of this happening bigger. The first time the screen mode of one of the vm's just changed to the left to right walking progress bar. This time I brought one of the windows to the front late in de bootprocess (the windows applying computer settings screen).
Signature · 5 lines
Hans -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1Quoted 4 lines
Quoted 3 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
Quoted 7 lines
Hans,
The oopses below point to the possibility of a hardware problem, similar to:
https://bugzilla.redhat.com/show_bug.cgi?id=480779
Can you please rule it out with memtest86?
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.
Signature · 60 lines
Hans by the way, the system is an asus m2npv-mx (latest bios) with 6G ram: bash-3.1# cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 107 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ stepping : 1 cpu MHz : 1000.000 cache size : 512 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch bogomips : 2004.31 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp tm stc 100mhzsteps ... bash-3.1# lspci 00:00.0 RAM memory: nVidia Corporation C51 Host Bridge (rev a2) 00:00.1 RAM memory: nVidia Corporation C51 Memory Controller 0 (rev a2) 00:00.2 RAM memory: nVidia Corporation C51 Memory Controller 1 (rev a2) 00:00.3 RAM memory: nVidia Corporation C51 Memory Controller 5 (rev a2) 00:00.4 RAM memory: nVidia Corporation C51 Memory Controller 4 (rev a2) 00:00.5 RAM memory: nVidia Corporation C51 Host Bridge (rev a2) 00:00.6 RAM memory: nVidia Corporation C51 Memory Controller 3 (rev a2) 00:00.7 RAM memory: nVidia Corporation C51 Memory Controller 2 (rev a2) 00:05.0 VGA compatible controller: nVidia Corporation C51PV [GeForce 6150] (rev a2) 00:09.0 RAM memory: nVidia Corporation MCP51 Host Bridge (rev a2) 00:0a.0 ISA bridge: nVidia Corporation MCP51 LPC Bridge (rev a3) 00:0a.1 SMBus: nVidia Corporation MCP51 SMBus (rev a3) 00:0a.2 RAM memory: nVidia Corporation MCP51 Memory Controller 0 (rev a3) 00:0b.0 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3) 00:0b.1 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3) 00:0d.0 IDE interface: nVidia Corporation MCP51 IDE (rev a1) 00:0e.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev a1) 00:0f.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev a1) 00:10.0 PCI bridge: nVidia Corporation MCP51 PCI Bridge (rev a2) 00:10.1 Audio device: nVidia Corporation MCP51 High Definition Audio (rev a2) 00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a3) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:09.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.Hans,
It would be helpful if you can capture a few more KVM oopses, then.
v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:
[ 167.829503] rmap_remove: ffff8801079a7a00 800000017c5c6047 1->BUG [ 167.829510] ------------[ cut here ]------------ [ 167.829513] kernel BUG at arch/x86/kvm/mmu.c:582! [ 167.829515] invalid opcode: 0000 [#1] SMP [ 167.829518] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource [ 167.829520] CPU 1 [ 167.829522] Modules linked in: [ 167.829526] Pid: 2908, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name [ 167.829528] RIP: 0010:[<ffffffff80216fff>] [<ffffffff80216fff>] rmap_remove+0xdf/0x200 [ 167.829536] RSP: 0018:ffff8801a13f19f8 EFLAGS: 00010292 [ 167.829538] RAX: 0000000000000049 RBX: 800000017c5c6047 RCX: ffffffff809a3b40 [ 167.829541] RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a30 [ 167.829543] RBP: ffff8801a13f1a28 R08: 000000000000ae32 R09: 00000000ffffffff [ 167.829545] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000017c5c6 [ 167.829548] R13: ffff8801079a7a00 R14: ffff8801094f8580 R15: ffff8801a121c000 [ 167.829551] FS: 000000004239a950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffd6000 [ 167.829553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 167.829556] CR2: 00007fd70812e540 CR3: 00000001a3fb8000 CR4: 00000000000006e0 [ 167.829558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 167.829560] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 167.829563] Process qemu-system-x86 (pid: 2908, threadinfo ffff8801a13f0000, task ffff8801ae9d1c20) [ 167.829565] Stack: [ 167.829566] ffff8801a13f1a28 0000000000035b00 0000000000035b2e 88e0000035b2e867 [ 167.829570] 0000000000000a00 ffff8801094f8580 ffff8801a13f1ac8 ffffffff8021ad8d [ 167.829574] 0000000000000000 ffff880100000000 000000000003633a 000000000017d5ea [ 167.829578] Call Trace: [ 167.829580] [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0 [ 167.829585] [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150 [ 167.829589] [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90 [ 167.829592] [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120 [ 167.829596] [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10 [ 167.829600] [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0 [ 167.829603] [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200 [ 167.829606] [<ffffffff8022821a>] ? handle_exit+0x14a/0x2c0 [ 167.829610] [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0 [ 167.829615] [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [ 167.829618] [<ffffffff8027cde9>] ? do_futex+0x689/0x9c0 [ 167.829623] [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [ 167.829626] [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13 [ 167.829630] [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [ 167.829633] [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90 [ 167.829638] [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0 [ 167.829641] [<ffffffff802d9562>] sys_ioctl+0x82/0xa0 [ 167.829645] [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b [ 167.829649] Code: 48 85 c0 0f 84 81 00 00 00 a8 01 75 3d 4c 39 e8 0f 84 f4 00 00 00 49 8b 55 00 4c 89 ee 48 c7 c7 f0 2e 7f 80 31 c0 e8 a1 29 04 00 <0f> 0b eb fe 4c 89 e7 e8 f5 2d ff ff eb 9b 48 89 c7 e8 cb 52 ff [ 167.829675] RIP [<ffffffff80216fff>] rmap_remove+0xdf/0x200 [ 167.829678] RSP <ffff8801a13f19f8> [ 167.829681] ---[ end trace bee56bd865cfd2e1 ]---
Quoted 11 lines · Hans de Bruin
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.Hans,
It would be helpful if you can capture a few more KVM oopses, then.
Quoted 5 lines
v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:
another one. This time the vm where booted sequentialy:
[ 253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0 [ 475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG [ 475.036549] ------------[ cut here ]------------ [ 475.036552] kernel BUG at arch/x86/kvm/mmu.c:576! [ 475.036555] invalid opcode: 0000 [#1] SMP [ 475.036558] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource [ 475.036560] CPU 1 [ 475.036562] Modules linked in: [ 475.036565] Pid: 2912, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name [ 475.036568] RIP: 0010:[<ffffffff80217071>] [<ffffffff80217071>] rmap_remove+0x151/0x200 [ 475.036576] RSP: 0018:ffff8801488759f8 EFLAGS: 00010292 [ 475.036578] RAX: 000000000000003a RBX: 0000000000000010 RCX: ffffffff809a3b40 [ 475.036580] RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a30 [ 475.036583] RBP: ffff880148875a28 R08: 000000000000ae73 R09: 00000000ffffffff [ 475.036585] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 475.036587] R13: ffff8800cdb913b0 R14: ffff8800c9c47580 R15: ffff88014889c000 [ 475.036590] FS: 0000000042131950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffae000 [ 475.036593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 475.036595] CR2: 00000000014bcc18 CR3: 0000000148846000 CR4: 00000000000006e0 [ 475.036597] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 475.036599] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 475.036602] Process qemu-system-x86 (pid: 2912, threadinfo ffff880148874000, task ffff8801afa06ae0) [ 475.036604] Stack: [ 475.036606] ffff880148875a28 0000000000000000 0000000053fbdb00 f880053fbdb00400 [ 475.036609] 00000000000003b0 ffff8800c9c47580 ffff880148875ac8 ffffffff8021ad8d [ 475.036613] 0000000000000000 ffff880100000100 00000000000252df 000000000012566f [ 475.036617] Call Trace: [ 475.036619] [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0 [ 475.036624] [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150 [ 475.036627] [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90 [ 475.036631] [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120 [ 475.036634] [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30 [ 475.036638] [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10 [ 475.036642] [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0 [ 475.036645] [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200 [ 475.036649] [<ffffffff8022821a>] ? handle_exit+0x14a/0x2c0 [ 475.036652] [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0 [ 475.036657] [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [ 475.036660] [<ffffffff8027c809>] ? do_futex+0xa9/0x9c0 [ 475.036665] [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [ 475.036668] [<ffffffff803ea2c1>] ? security_file_permission+0x11/0x20 [ 475.036672] [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [ 475.036676] [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90 [ 475.036680] [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0 [ 475.036684] [<ffffffff8027d1a4>] ? sys_futex+0x84/0x130 [ 475.036687] [<ffffffff802d9562>] sys_ioctl+0x82/0xa0 [ 475.036691] [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b [ 475.036695] Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7 eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 d0 2e 7f 80 e8 2f 29 04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84 [ 475.036721] RIP [<ffffffff80217071>] rmap_remove+0x151/0x200 [ 475.036724] RSP <ffff8801488759f8> [ 475.036727] ---[ end trace 5a588ebf5d2bfd50 ]---
bash-3.1$
bash-3.1$
--
Hans
Quoted 12 lines · Hans de Bruin
Quoted 11 lines · Hans de Bruin
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.Hans,
It would be helpful if you can capture a few more KVM oopses, then.
Quoted 8 lines
Quoted 5 lines
v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:another one. This time the vm where booted sequentialy:
just for fun single winxp over nfs and wireless (really slow):
[ 978.108416] kvm: emulating exchange as write [ 1083.639299] CE: hpet increasing min_delta_ns to 15000 nsec [ 1188.704021] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 [ 1188.704028] IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 [ 1188.704036] PGD 1a0c63067 PUD 1a0ddd067 PMD 0 [ 1188.704040] Oops: 0000 [#1] SMP [ 1188.704043] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource [ 1188.704046] CPU 1 [ 1188.704047] Modules linked in: [ 1188.704051] Pid: 2985, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name [ 1188.704054] RIP: 0010:[<ffffffff80216b4c>] [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 [ 1188.704059] RSP: 0018:ffff8801a0dc1918 EFLAGS: 00010296 [ 1188.704061] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001 [ 1188.704063] RDX: ffffe20000000000 RSI: ffff880195717140 RDI: 0000000000000000 [ 1188.704066] RBP: ffff8801a0dc1928 R08: 0000000000000001 R09: 0000000000000001 [ 1188.704068] R10: 0000000000000002 R11: 0000000000000000 R12: ffff880195717140 [ 1188.704070] R13: ffff88019542ab00 R14: 0000000000000028 R15: ffff8801a0cc0000 [ 1188.704073] FS: 0000000040cd7950(0063) GS:ffff88002804d000(0000) knlGS:0000000000000000 [ 1188.704076] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1188.704078] CR2: 0000000000000058 CR3: 00000001a3d48000 CR4: 00000000000006e0 [ 1188.704080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1188.704083] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1188.704086] Process qemu-system-x86 (pid: 2985, threadinfo ffff8801a0dc0000, task ffff8801a0d8d460) [ 1188.704088] Stack: [ 1188.704089] ffff8801acd60000 0000000000000008 ffff8801a0dc1ab8 ffffffff80217323 [ 1188.704093] ffff8801a0cc0000 0000000000000001 ffff8801955baa50 ffffffff00000000 [ 1188.704097] 0000000000000001 00000000fee0017b ffff8801a0dc19f8 ffffffff8021a04f [ 1188.704101] Call Trace: [ 1188.704103] [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0 [ 1188.704107] [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320 [ 1188.704111] [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60 [ 1188.704115] [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70 [ 1188.704119] [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0 [ 1188.704124] [<ffffffff80217594>] __kvm_mmu_free_some_pages+0x24/0x40 [ 1188.704127] [<ffffffff8021b55d>] paging64_page_fault+0x40d/0x420 [ 1188.704131] [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30 [ 1188.704135] [<ffffffff8021ca75>] ? x86_decode_insn+0xc45/0xf10 [ 1188.704139] [<ffffffff8020f98d>] ? emulate_instruction+0x1cd/0x2f0 [ 1188.704143] [<ffffffff8021919c>] kvm_mmu_page_fault+0x1c/0x90 [ 1188.704147] [<ffffffff80226e6f>] pf_interception+0x7f/0x190 [ 1188.704151] [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70 [ 1188.704155] [<ffffffff8022821a>] handle_exit+0x14a/0x2c0 [ 1188.704158] [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70 [ 1188.704161] [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0 [ 1188.704165] [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910 [ 1188.704169] [<ffffffff8027cde9>] ? do_futex+0x689/0x9c0 [ 1188.704174] [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790 [ 1188.704177] [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50 [ 1188.704181] [<ffffffff8024eb36>] ? set_next_entity+0x26/0x50 [ 1188.704184] [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90 [ 1188.704189] [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0 [ 1188.704192] [<ffffffff802d9562>] sys_ioctl+0x82/0xa0 [ 1188.704196] [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b [ 1188.704200] Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f [ 1188.704226] RIP [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100 [ 1188.704230] RSP <ffff8801a0dc1918> [ 1188.704232] CR2: 0000000000000058 [ 1188.704234] ---[ end trace 35a9bc3a55fc3772 ]---
Signature · 1 line
Hans
Quoted 21 lines · Hans de Bruin
Quoted 14 lines · Hans de Bruin
Quoted 11 lines · Marcelo Tosatti
Quoted 5 lines · Hans de Bruin
I ran memtest for 11 hours and it completed 4.7 passes with no problems. But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.
Hans,
It would be helpful if you can capture a few more KVM oopses, then.
Quoted 5 lines
v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:
Quoted 5 lines
another one. This time the vm where booted sequentialy:
[ 253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0 [ 475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
^^^
So 0x10 is 1 bit different from 0x00, which would be the no present entry for AMD.
Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.
Quoted 20 lines · Marcelo Tosatti
Quoted 15 lines · Hans de Bruin
Quoted 6 lines · Hans de Bruin
Quoted 5 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.Hans,
Quoted 8 lines
Quoted 2 lines
It would be helpful if you can capture a few more KVM oopses, then.
v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:Quoted 4 lines
another one. This time the vm where booted sequentialy:
[ 253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0 [ 475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
Quoted 8 lines
^^^So
0x10is 1 bit different from0x00, which would be the no present entry for AMD.Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.
That is still on my to do list.
Signature · 1 line
Hans
Quoted 5 lines · Marcelo Tosatti
Quoted 5 lines
another one. This time the vm where booted sequentialy:
[ 253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0 [ 475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
Quoted 8 lines
^^^So
0x10is 1 bit different from0x00, which would be the no present entry for AMD.Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.
It's actually the PCD bit, not that it affects your analysis. Strange that we see this pattern (1 bit differences on sptes) on both AMD and Intel. Very suspicious.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Marcelo Tosatti
Quoted 6 lines
another one. This time the vm where booted sequentialy:
[ 253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0 [ 475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG^^^So
0x10is 1 bit different from0x00, which would be the no present entry for AMD.
Quoted 7 lines
Quoted 3 lines
Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.
It's actually the PCD bit, not that it affects your analysis. Strange that we see this pattern (1 bit differences on sptes) on both AMD and Intel. Very suspicious.
On Intel there was confirmation (through memtest86) that it was memory failure. The pattern was:
0xff7ffffffffff001
Quoted 34 lines · Hans de Bruin
Quoted 33 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1Quoted 5 lines
Quoted 4 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
Hans,
The oopses below point to the possibility of a hardware problem, similar to:
Quoted 7 lines
Quoted 1 line
Can you please rule it out with memtest86?
I ran memtest for 11 hours and it completed 4.7 passes with no problems. But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.
Hans:
There is a memory test designed to stress disk/chipset/dma interaction. I made an implementation of it on autotest, the test module is called dma_memtest:
http://autotest.kernel.org/browser/trunk/client/tests/dma_memtest/dma_memtest.py
The work that originated this autotest implementation can be found on:
http://people.redhat.com/dledford/memtest.shtml
So you can choose either to run the autotest version or the shell script provided by Doug. Try running either of them, we might see interesting results.
Regards,
Signature · 3 lines
Lucas Meneghel Rodrigues Software Engineer (QE) Red Hat - Emerging Technologies
Quoted 36 lines · Lucas Meneghel Rodrigues
Quoted 27 lines · Hans de Bruin
Quoted 26 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1Quoted 4 lines
Quoted 3 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
Hans,
Quoted 7 lines
Quoted 5 lines
The oopses below point to the possibility of a hardware problem, similar to:
https://bugzilla.redhat.com/show_bug.cgi?id=480779
Can you please rule it out with memtest86?
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.
Quoted 7 lines
Hans:
There is a memory test designed to stress disk/chipset/dma interaction. I made an implementation of it on autotest, the test module is called
dma_memtest:http://autotest.kernel.org/browser/trunk/client/tests/dma_memtest/dma_memtest.py
[09:09:47 INFO ] Test finished after 1 iterations.
Memory test passed.
[09:09:48 DEBUG] Running 'grep MemTotal /proc/meminfo' [09:09:48 DEBUG] Running 'rpm -qa'
[09:09:49 INFO ] GOOD dma_memtest dma_memtest timestamp=1243408189 localtime=May 27 09:09:49 completed successfully
[09:09:49 DEBUG] Persistent state variable __group_level now set to 1
[09:09:49 INFO ] END GOOD dma_memtest dma_memtest timestamp=1243408189 localtime=May 27 09:09:49
[09:09:49 DEBUG] Dropping caches
[09:09:49 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'echo 3 > /proc/sys/vm/drop_caches' [09:09:52 DEBUG] Persistent state variable __group_level now set to 0
[09:09:52 INFO ] END GOOD ---- ---- timestamp=1243408192 localtime=May 27 09:09:52
Well that looks good. The web page talks about forcing the system to swap. That never happend swap usage is still 0 bytes. I installed autotest on the same lv (3 disk stripe) as the vmdisks.
Signature · 1 line
Hans
Quoted 19 lines · Hans de Bruin
[09:09:47 INFO ] Test finished after 1 iterations.
Memory test passed.
[09:09:48 DEBUG] Running 'grep MemTotal/proc/meminfo' [09:09:48 DEBUG] Running 'rpm -qa'
[09:09:49 INFO ] GOODdma_memtestdma_memtest
timestamp=1243408189 localtime=May 27 09:09:49 completed
successfully
[09:09:49 DEBUG] Persistent state variable__group_levelnow set to 1
[09:09:49 INFO ] END GOODdma_memtestdma_memtest
timestamp=1243408189 localtime=May 27 09:09:49
[09:09:49 DEBUG] Dropping caches
[09:09:49 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'echo 3 >/proc/sys/vm/drop_caches' [09:09:52 DEBUG] Persistent state variable__group_levelnow set to 0
[09:09:52 INFO ] END GOOD ---- ---- timestamp=1243408192
localtime=May 27 09:09:52
Quoted 3 lines
Well that looks good. The web page talks about forcing the system to swap. That never happend swap usage is still 0 bytes. I installed autotest on the same lv (3 disk stripe) as the vmdisks.
Interesting, about that I made some tests and just realized that in some cases both mine and the original shell implementation are failing on forcing the system to go to swap (the tests I made when the test was firstly implemented did manage to make the machines swap, but in conditions where the system had a significantly higher initial memory usage). I will work on a better heuristic to force swap.
Thanks for pointing this out,
Signature · 3 lines
Lucas Meneghel Rodrigues Software Engineer (QE) Red Hat - Emerging Technologies
Quoted 29 lines · Hans de Bruin
Quoted 28 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin
Staring to vms simultaneously end in crash
linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcorevm's like:
#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1Quoted 5 lines
Quoted 4 lines
qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge
another dmesg:
Hans,
Quoted 9 lines
Quoted 5 lines
The oopses below point to the possibility of a hardware problem, similar to:
https://bugzilla.redhat.com/show_bug.cgi?id=480779
Can you please rule it out with memtest86?
I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.
And now summer has arived, and the temperatures are going up. Instead of just the kvm troubles, its now impossible to compile a kernel. I am going to replace my hardware.
Signature · 1 line
Hans
Quoted 1 line · Hans de Bruin
And now summer has arived, and the temperatures are going up. Instead of just the kvm troubles, its now impossible to compile a kernel. I am going to replace my hardware.
Apparently the virtualization extensions stress hardware. Maybe it's the long microcode sequences.
Signature · 1 line
error compiling committee.c: too many arguments to function