Skip to contentSkip to conversation

just a dump

DISCUSSION Hans de Bruin 4 participants 16 messages May 12, 2009 – Jul 6, 2009 1 message not in the archive KVM

lore
Share & tools
lore
#0 lore raw

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

dmesg:
device tap0 entered promiscuous mode
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap1: no IPv6 routers present
tap0: no IPv6 routers present
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0

BUG: unable to handle kernel NULL pointer dereference at (null)

device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap1: no IPv6 routers present
tap0: no IPv6 routers present
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2935: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60
PGD 1a0c4b067 PUD 1a0c4a067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
CPU 0
Modules linked in:
Pid: 2945, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System Product Name
RIP: 0010:[<ffffffff80216ee2>]  [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60
RSP: 0018:ffff8801a0f979d8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000e1000 RSI: af7dc78941006c5f RDI: ffff8801a0d90000
RBP: ffff8801a0f979e8 R08: 0000000000000006 R09: 0000000000000006
R10: 0000000000000000 R11: 0000000000000000 R12: af7dc78941006c5f
R13: ffff8801908b1180 R14: ffff88019e778c60 R15: ffff8801a0d90000
FS:  0000000041591950(0063) GS:ffff880028034000(0000) knlGS:fffff800017ce500
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000001a3d7c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 2945, threadinfo ffff8801a0f96000, task ffff8801afb12d00)
Stack:
 0000000000000010 0000000000000000 ffff8801a0f97a28 ffffffff80216fce
 ffff8801a0f97a28 af7dc78941006c5f 0000000000000000 0000000000000000
 0000000000000180 ffff88019e778c60 ffff8801a0f97ac8 ffffffff8021ad8d
Call Trace:
 [<ffffffff80216fce>] rmap_remove+0xae/0x200
 [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0
 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30
 [<ffffffff80216edb>] ? gfn_to_rmap+0x1b/0x60
 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150
 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90
 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120
 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0
 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200
 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0
 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0
 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0
 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13
 [<ffffffff8022b9ce>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90
 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0
 [<ffffffff8027d154>] ? sys_futex+0x84/0x130
 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0
 [<ffffffff802cc08b>] ? sys_lseek+0x7b/0x80
 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
Code: 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 89 d3 4c 89 64 24 08 49 89 f4 e8 a5 36 ff ff 85 db 48 89 c1 75 1e <4c> 2b 20 4a 8d 14 e5 00 00 00 00 48 03 50 18 48 8b 1c 24 4c 8b
RIP  [<ffffffff80216ee2>] gfn_to_rmap+0x22/0x60
 RSP <ffff8801a0f979d8>
CR2: 0000000000000000
---[ end trace 5b490dc3e478a040 ]---

kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2934: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
CE: hpet increasing min_delta_ns to 15000 nsec

one vm was still running.

Signature · 1 line
Hans
#1 lore raw
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 3 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

device tap0 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap0: no IPv6 routers present
tap1: no IPv6 routers present
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
rmap_remove: ffff880100de5500 8 0->BUG

------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:576!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
CPU 1
Modules linked in:
Pid: 2925, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System Product Name
RIP: 0010:[<ffffffff80217071>]  [<ffffffff80217071>] rmap_remove+0x151/0x200
RSP: 0018:ffff8801a0d379f8  EFLAGS: 00010292
RAX: 000000000000002a RBX: 0000000000000008 RCX: ffffffff809a3b40
RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a34
RBP: ffff8801a0d37a28 R08: 0000000000008777 R09: 00000000ffffffff
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880100de5500 R14: ffff880101e23580 R15: ffff8801a0e1c000
FS:  000000004270d950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffaa000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000014a8c18 CR3: 00000001a0c62000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 2925, threadinfo ffff8801a0d36000, task ffff8801af3605a0)
Stack:
 ffff8801a0d37a28 0000000000000000 0000000000000000 0000000000000000
 0000000000000500 ffff880101e23580 ffff8801a0d37ac8 ffffffff8021ad8d
 0000000000000000 ffff880100000000 000000000003020d 000000000016e772
Call Trace:
 [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0
 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150
 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90
 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120
 [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30
 [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10
 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0
 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200
 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0
 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0
 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0
 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13
 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90
 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0
 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0
 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7 eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 b8 2e 7f 80 e8 1f 29
04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84
RIP  [<ffffffff80217071>] rmap_remove+0x151/0x200
 RSP <ffff8801a0d379f8>
---[ end trace c11385df745a1fea ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
PGD 1a0ca8067 PUD 1a0ca9067 PMD 0
Oops: 0000 [#2] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
CPU 0
Modules linked in:
Pid: 2926, comm: qemu-system-x86 Tainted: G      D    2.6.30-rc5 #3 System Product Name
RIP: 0010:[<ffffffff80216b4c>]  [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
RSP: 0018:ffff8801a0da57a8  EFLAGS: 00010292
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002b
RDX: ffffe20000000000 RSI: ffff8800ccac0220 RDI: 0000000000000000
RBP: ffff8801a0da57b8 R08: 000000000000006a R09: ffff8800ccd85e70
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800ccac0220
R13: ffff8800ccd85dc0 R14: 0000000000000044 R15: ffff8801a0db0000
FS:  0000000040fbc950(0063) GS:ffff880028034000(0000) knlGS:000007fffffd5000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 00000001a0c63000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 2926, threadinfo ffff8801a0da4000, task ffff8801ae971c20)
Stack:
 ffff8800ccd85590 000000000000007a ffff8801a0da5948 ffffffff80217323
 ffff8801a0da5808 0000000000000056 ffff8800ccd85dc0 ffffe20000000000
 ffff8801030f8160 0000000000000003 ffff880103f87000 ffffffff000001b8
Call Trace:
 [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0
 [<ffffffff8020a207>] ? mark_page_dirty+0x27/0x60
 [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0
 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30
 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30
 [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60
 [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70
 [<ffffffff8021993c>] kvm_mmu_pte_write+0x72c/0x910
 [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320
 [<ffffffff8020a31c>] ? kvm_write_guest_page+0x7c/0x80
 [<ffffffff8020fb0d>] emulator_write_phys+0x4d/0x70
 [<ffffffff80211785>] emulator_write_emulated_onepage+0x95/0x120
 [<ffffffff80211880>] emulator_write_emulated+0x70/0x90
 [<ffffffff8021d11e>] x86_emulate_insn+0x3de/0x4d30
 [<ffffffff8021bcef>] ? decode_register_operand+0x8f/0x100
 [<ffffffff8021c50c>] ? x86_decode_insn+0x6dc/0xf10
 [<ffffffff8020e710>] ? kvm_find_cpuid_entry+0xf0/0x110
 [<ffffffff8020f91f>] emulate_instruction+0x15f/0x2f0
 [<ffffffff802191da>] kvm_mmu_page_fault+0x5a/0x90
 [<ffffffff80226e7f>] pf_interception+0x7f/0x190
 [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70
 [<ffffffff8022822a>] handle_exit+0x14a/0x2c0
 [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0
 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
 [<ffffffff8026e680>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
 [<ffffffff803ea171>] ? security_file_permission+0x11/0x20
 [<ffffffff802cba0f>] ? do_readv_writev+0x14f/0x1d0
 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90
 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0
 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0
 [<ffffffff802cc4c1>] ? sys_writev+0x81/0x90
 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f
RIP  [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
 RSP <ffff8801a0da57a8>
CR2: 0000000000000058
---[ end trace c11385df745a1feb ]---

Starting with a clear disk cach makes the change of this happening bigger. The first time the screen mode of one of the vm's just changed to the left to right walking progress bar. This time I brought one of the windows to the front late in de bootprocess (the windows applying computer settings screen).

Signature · 1 line
Hans
#2 lore raw
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 5 lines
Quoted 4 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

Hans,

The oopses below point to the possibility of a hardware problem, similar to:

https://bugzilla.redhat.com/show_bug.cgi?id=480779

Can you please rule it out with memtest86?

Quoted 152 lines

device tap0 entered promiscuous mode
br1: topology change detected, propagating
br1: port 1(tap0) entering forwarding state
device tap1 entered promiscuous mode
br1: topology change detected, propagating
br1: port 2(tap1) entering forwarding state
tap0: no IPv6 routers present
tap1: no IPv6 routers present
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2915: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010001 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010002 data 0x0
kvm: 2914: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
rmap_remove: ffff880100de5500 8 0->BUG

------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:576!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
CPU 1
Modules linked in:
Pid: 2925, comm: qemu-system-x86 Not tainted 2.6.30-rc5 #3 System  
Product Name
RIP: 0010:[<ffffffff80217071>]  [<ffffffff80217071>] rmap_remove+0x151/0x200
RSP: 0018:ffff8801a0d379f8  EFLAGS: 00010292
RAX: 000000000000002a RBX: 0000000000000008 RCX: ffffffff809a3b40
RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a34
RBP: ffff8801a0d37a28 R08: 0000000000008777 R09: 00000000ffffffff
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880100de5500 R14: ffff880101e23580 R15: ffff8801a0e1c000
FS:  000000004270d950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffaa000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000014a8c18 CR3: 00000001a0c62000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 2925, threadinfo ffff8801a0d36000, task  
ffff8801af3605a0)
Stack:
 ffff8801a0d37a28 0000000000000000 0000000000000000 0000000000000000
 0000000000000500 ffff880101e23580 ffff8801a0d37ac8 ffffffff8021ad8d
 0000000000000000 ffff880100000000 000000000003020d 000000000016e772
Call Trace:
 [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0
 [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150
 [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90
 [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120
 [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30
 [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10
 [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0
 [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200
 [<ffffffff8022822a>] ? handle_exit+0x14a/0x2c0
 [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0
 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
 [<ffffffff8027cda9>] ? do_futex+0x679/0x9a0
 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
 [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13
 [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90
 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0
 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0
 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7  

eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 b8 2e 7f 80 e8 1f 29
04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84 RIP [<ffffffff80217071>] rmap_remove+0x151/0x200
RSP <ffff8801a0d379f8>

---[ end trace c11385df745a1fea ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
PGD 1a0ca8067 PUD 1a0ca9067 PMD 0
Oops: 0000 [#2] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
CPU 0
Modules linked in:
Pid: 2926, comm: qemu-system-x86 Tainted: G      D    2.6.30-rc5 #3  
System Product Name
RIP: 0010:[<ffffffff80216b4c>]  [<ffffffff80216b4c>]  
mmu_page_remove_parent_pte+0xc/0x100
RSP: 0018:ffff8801a0da57a8  EFLAGS: 00010292
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002b
RDX: ffffe20000000000 RSI: ffff8800ccac0220 RDI: 0000000000000000
RBP: ffff8801a0da57b8 R08: 000000000000006a R09: ffff8800ccd85e70
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800ccac0220
R13: ffff8800ccd85dc0 R14: 0000000000000044 R15: ffff8801a0db0000
FS:  0000000040fbc950(0063) GS:ffff880028034000(0000) knlGS:000007fffffd5000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 00000001a0c63000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 2926, threadinfo ffff8801a0da4000, task  
ffff8801ae971c20)
Stack:
 ffff8800ccd85590 000000000000007a ffff8801a0da5948 ffffffff80217323
 ffff8801a0da5808 0000000000000056 ffff8800ccd85dc0 ffffe20000000000
 ffff8801030f8160 0000000000000003 ffff880103f87000 ffffffff000001b8
Call Trace:
 [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0
 [<ffffffff8020a207>] ? mark_page_dirty+0x27/0x60
 [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0
 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30
 [<ffffffff8020a59c>] ? gfn_to_memslot+0x1c/0x30
 [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60
 [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70
 [<ffffffff8021993c>] kvm_mmu_pte_write+0x72c/0x910
 [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320
 [<ffffffff8020a31c>] ? kvm_write_guest_page+0x7c/0x80
 [<ffffffff8020fb0d>] emulator_write_phys+0x4d/0x70
 [<ffffffff80211785>] emulator_write_emulated_onepage+0x95/0x120
 [<ffffffff80211880>] emulator_write_emulated+0x70/0x90
 [<ffffffff8021d11e>] x86_emulate_insn+0x3de/0x4d30
 [<ffffffff8021bcef>] ? decode_register_operand+0x8f/0x100
 [<ffffffff8021c50c>] ? x86_decode_insn+0x6dc/0xf10
 [<ffffffff8020e710>] ? kvm_find_cpuid_entry+0xf0/0x110
 [<ffffffff8020f91f>] emulate_instruction+0x15f/0x2f0
 [<ffffffff802191da>] kvm_mmu_page_fault+0x5a/0x90
 [<ffffffff80226e7f>] pf_interception+0x7f/0x190
 [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70
 [<ffffffff8022822a>] handle_exit+0x14a/0x2c0
 [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0
 [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
 [<ffffffff8026e680>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
 [<ffffffff803ea171>] ? security_file_permission+0x11/0x20
 [<ffffffff802cba0f>] ? do_readv_writev+0x14f/0x1d0
 [<ffffffff802d8f31>] vfs_ioctl+0x31/0x90
 [<ffffffff802d9281>] do_vfs_ioctl+0x2f1/0x4e0
 [<ffffffff802d94f2>] sys_ioctl+0x82/0xa0
 [<ffffffff802cc4c1>] ? sys_writev+0x81/0x90
 [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66  

2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f RIP [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
RSP <ffff8801a0da57a8>
CR2: 0000000000000058

---[ end trace c11385df745a1feb ]---

Starting with a clear disk cach makes the change of this happening bigger. The first time the screen mode of one of the vm's just changed to the left to right walking progress bar. This time I brought one of the windows to the front late in de bootprocess (the windows applying computer settings screen).

Signature · 5 lines
Hans
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
Quoted 25 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 4 lines
Quoted 3 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

Quoted 7 lines

Hans,

The oopses below point to the possibility of a hardware problem, similar to:

https://bugzilla.redhat.com/show_bug.cgi?id=480779

Can you please rule it out with memtest86?

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Signature · 60 lines
Hans

by the way, the system is an asus m2npv-mx (latest bios) with 6G ram:

bash-3.1# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 107
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4000+
stepping        : 1
cpu MHz         : 1000.000
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
bogomips        : 2004.31
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps
...


bash-3.1# lspci
00:00.0 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.1 RAM memory: nVidia Corporation C51 Memory Controller 0 (rev a2)
00:00.2 RAM memory: nVidia Corporation C51 Memory Controller 1 (rev a2)
00:00.3 RAM memory: nVidia Corporation C51 Memory Controller 5 (rev a2)
00:00.4 RAM memory: nVidia Corporation C51 Memory Controller 4 (rev a2)
00:00.5 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.6 RAM memory: nVidia Corporation C51 Memory Controller 3 (rev a2)
00:00.7 RAM memory: nVidia Corporation C51 Memory Controller 2 (rev a2)
00:05.0 VGA compatible controller: nVidia Corporation C51PV [GeForce 6150] (rev a2)
00:09.0 RAM memory: nVidia Corporation MCP51 Host Bridge (rev a2)
00:0a.0 ISA bridge: nVidia Corporation MCP51 LPC Bridge (rev a3)
00:0a.1 SMBus: nVidia Corporation MCP51 SMBus (rev a3)
00:0a.2 RAM memory: nVidia Corporation MCP51 Memory Controller 0 (rev a3)
00:0b.0 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0b.1 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0d.0 IDE interface: nVidia Corporation MCP51 IDE (rev a1)
00:0e.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev a1)
00:0f.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev a1)
00:10.0 PCI bridge: nVidia Corporation MCP51 PCI Bridge (rev a2)
00:10.1 Audio device: nVidia Corporation MCP51 High Definition Audio (rev a2)
00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:09.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans,

It would be helpful if you can capture a few more KVM oopses, then.

v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:

[  167.829503] rmap_remove:  ffff8801079a7a00 800000017c5c6047 1->BUG
[  167.829510] ------------[ cut here ]------------
[  167.829513] kernel BUG at arch/x86/kvm/mmu.c:582!
[  167.829515] invalid opcode: 0000 [#1] SMP
[  167.829518] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
[  167.829520] CPU 1
[  167.829522] Modules linked in:
[  167.829526] Pid: 2908, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name
[  167.829528] RIP: 0010:[<ffffffff80216fff>]  [<ffffffff80216fff>] rmap_remove+0xdf/0x200
[  167.829536] RSP: 0018:ffff8801a13f19f8  EFLAGS: 00010292
[  167.829538] RAX: 0000000000000049 RBX: 800000017c5c6047 RCX: ffffffff809a3b40
[  167.829541] RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a30
[  167.829543] RBP: ffff8801a13f1a28 R08: 000000000000ae32 R09: 00000000ffffffff
[  167.829545] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000017c5c6
[  167.829548] R13: ffff8801079a7a00 R14: ffff8801094f8580 R15: ffff8801a121c000
[  167.829551] FS:  000000004239a950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffd6000
[  167.829553] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  167.829556] CR2: 00007fd70812e540 CR3: 00000001a3fb8000 CR4: 00000000000006e0
[  167.829558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  167.829560] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  167.829563] Process qemu-system-x86 (pid: 2908, threadinfo ffff8801a13f0000, task ffff8801ae9d1c20)
[  167.829565] Stack:
[  167.829566]  ffff8801a13f1a28 0000000000035b00 0000000000035b2e 88e0000035b2e867
[  167.829570]  0000000000000a00 ffff8801094f8580 ffff8801a13f1ac8 ffffffff8021ad8d
[  167.829574]  0000000000000000 ffff880100000000 000000000003633a 000000000017d5ea
[  167.829578] Call Trace:
[  167.829580]  [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0
[  167.829585]  [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150
[  167.829589]  [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90
[  167.829592]  [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120
[  167.829596]  [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10
[  167.829600]  [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0
[  167.829603]  [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200
[  167.829606]  [<ffffffff8022821a>] ? handle_exit+0x14a/0x2c0
[  167.829610]  [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0
[  167.829615]  [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
[  167.829618]  [<ffffffff8027cde9>] ? do_futex+0x689/0x9c0
[  167.829623]  [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
[  167.829626]  [<ffffffff8022b88e>] ? common_interrupt+0xe/0x13
[  167.829630]  [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
[  167.829633]  [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90
[  167.829638]  [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0
[  167.829641]  [<ffffffff802d9562>] sys_ioctl+0x82/0xa0
[  167.829645]  [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
[  167.829649] Code: 48 85 c0 0f 84 81 00 00 00 a8 01 75 3d 4c 39 e8 0f 84 f4 00 00 00 49 8b 55 00 4c 89 ee 48 c7 c7 f0 2e 7f 80 31 c0 e8 a1 29 04 00 <0f> 0b eb fe 4c 89 e7 e8 f5 2d ff ff eb 9b 48 89 c7 e8 cb 52 ff
[  167.829675] RIP  [<ffffffff80216fff>] rmap_remove+0xdf/0x200
[  167.829678]  RSP <ffff8801a13f19f8>
[  167.829681] ---[ end trace bee56bd865cfd2e1 ]---
#6 lore raw
Quoted 11 lines · Hans de Bruin
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans,

It would be helpful if you can capture a few more KVM oopses, then.

Quoted 5 lines

v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:

another one. This time the vm where booted sequentialy:

[  253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003 data 0x0
[  475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
[  475.036549] ------------[ cut here ]------------
[  475.036552] kernel BUG at arch/x86/kvm/mmu.c:576!
[  475.036555] invalid opcode: 0000 [#1] SMP
[  475.036558] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
[  475.036560] CPU 1
[  475.036562] Modules linked in:
[  475.036565] Pid: 2912, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name
[  475.036568] RIP: 0010:[<ffffffff80217071>]  [<ffffffff80217071>] rmap_remove+0x151/0x200
[  475.036576] RSP: 0018:ffff8801488759f8  EFLAGS: 00010292
[  475.036578] RAX: 000000000000003a RBX: 0000000000000010 RCX: ffffffff809a3b40
[  475.036580] RDX: ffff88002804d000 RSI: 0000000000000046 RDI: ffffffff809a3a30
[  475.036583] RBP: ffff880148875a28 R08: 000000000000ae73 R09: 00000000ffffffff
[  475.036585] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  475.036587] R13: ffff8800cdb913b0 R14: ffff8800c9c47580 R15: ffff88014889c000
[  475.036590] FS:  0000000042131950(0063) GS:ffff88002804d000(0000) knlGS:000007fffffae000
[  475.036593] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  475.036595] CR2: 00000000014bcc18 CR3: 0000000148846000 CR4: 00000000000006e0
[  475.036597] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  475.036599] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  475.036602] Process qemu-system-x86 (pid: 2912, threadinfo ffff880148874000, task ffff8801afa06ae0)
[  475.036604] Stack:
[  475.036606]  ffff880148875a28 0000000000000000 0000000053fbdb00 f880053fbdb00400
[  475.036609]  00000000000003b0 ffff8800c9c47580 ffff880148875ac8 ffffffff8021ad8d
[  475.036613]  0000000000000000 ffff880100000100 00000000000252df 000000000012566f
[  475.036617] Call Trace:
[  475.036619]  [<ffffffff8021ad8d>] paging64_sync_page+0x9d/0x1a0
[  475.036624]  [<ffffffff80218825>] ? rmap_write_protect+0xd5/0x150
[  475.036627]  [<ffffffff8021890b>] kvm_sync_page+0x6b/0x90
[  475.036631]  [<ffffffff8021a1ad>] mmu_sync_children+0xcd/0x120
[  475.036634]  [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30
[  475.036638]  [<ffffffff8021c242>] ? x86_decode_insn+0x412/0xf10
[  475.036642]  [<ffffffff8021a2c2>] mmu_sync_roots+0xc2/0xd0
[  475.036645]  [<ffffffff8021a658>] kvm_mmu_load+0x138/0x200
[  475.036649]  [<ffffffff8022821a>] ? handle_exit+0x14a/0x2c0
[  475.036652]  [<ffffffff80213873>] kvm_arch_vcpu_ioctl_run+0x863/0xaa0
[  475.036657]  [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
[  475.036660]  [<ffffffff8027c809>] ? do_futex+0xa9/0x9c0
[  475.036665]  [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
[  475.036668]  [<ffffffff803ea2c1>] ? security_file_permission+0x11/0x20
[  475.036672]  [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
[  475.036676]  [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90
[  475.036680]  [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0
[  475.036684]  [<ffffffff8027d1a4>] ? sys_futex+0x84/0x130
[  475.036687]  [<ffffffff802d9562>] sys_ioctl+0x82/0xa0
[  475.036691]  [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
[  475.036695] Code: 04 75 e7 48 8b 47 20 49 89 fb 48 85 c0 0f 84 b7 00 00 00 48 89 c7 eb d0 49 8b 55 00 4c 89 ee 48 c7 c7 d0 2e 7f 80 e8 2f 29 04 00 <0f> 0b eb fe 48 8b 4f 18 48 85 c9 0f 94 c2 83 fe 02 0f 9e c0 84
[  475.036721] RIP  [<ffffffff80217071>] rmap_remove+0x151/0x200
[  475.036724]  RSP <ffff8801488759f8>
[  475.036727] ---[ end trace 5a588ebf5d2bfd50 ]---

bash-3.1$
bash-3.1$

--
Hans

#7 lore raw
Quoted 12 lines · Hans de Bruin
Quoted 11 lines · Hans de Bruin
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans,

It would be helpful if you can capture a few more KVM oopses, then.

Quoted 8 lines
Quoted 5 lines

v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:

another one. This time the vm where booted sequentialy:

just for fun single winxp over nfs and wireless (really slow):

[  978.108416] kvm: emulating exchange as write
[ 1083.639299] CE: hpet increasing min_delta_ns to 15000 nsec
[ 1188.704021] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
[ 1188.704028] IP: [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
[ 1188.704036] PGD 1a0c63067 PUD 1a0ddd067 PMD 0
[ 1188.704040] Oops: 0000 [#1] SMP
[ 1188.704043] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/0000:01:09.0/resource
[ 1188.704046] CPU 1
[ 1188.704047] Modules linked in:
[ 1188.704051] Pid: 2985, comm: qemu-system-x86 Not tainted 2.6.30-rc6 #5 System Product Name
[ 1188.704054] RIP: 0010:[<ffffffff80216b4c>]  [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
[ 1188.704059] RSP: 0018:ffff8801a0dc1918  EFLAGS: 00010296
[ 1188.704061] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[ 1188.704063] RDX: ffffe20000000000 RSI: ffff880195717140 RDI: 0000000000000000
[ 1188.704066] RBP: ffff8801a0dc1928 R08: 0000000000000001 R09: 0000000000000001
[ 1188.704068] R10: 0000000000000002 R11: 0000000000000000 R12: ffff880195717140
[ 1188.704070] R13: ffff88019542ab00 R14: 0000000000000028 R15: ffff8801a0cc0000
[ 1188.704073] FS:  0000000040cd7950(0063) GS:ffff88002804d000(0000) knlGS:0000000000000000
[ 1188.704076] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1188.704078] CR2: 0000000000000058 CR3: 00000001a3d48000 CR4: 00000000000006e0
[ 1188.704080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1188.704083] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1188.704086] Process qemu-system-x86 (pid: 2985, threadinfo ffff8801a0dc0000, task ffff8801a0d8d460)
[ 1188.704088] Stack:
[ 1188.704089]  ffff8801acd60000 0000000000000008 ffff8801a0dc1ab8 ffffffff80217323
[ 1188.704093]  ffff8801a0cc0000 0000000000000001 ffff8801955baa50 ffffffff00000000
[ 1188.704097]  0000000000000001 00000000fee0017b ffff8801a0dc19f8 ffffffff8021a04f
[ 1188.704101] Call Trace:
[ 1188.704103]  [<ffffffff80217323>] kvm_mmu_zap_page+0x153/0x3a0
[ 1188.704107]  [<ffffffff8021a04f>] ? paging64_walk_addr+0x28f/0x320
[ 1188.704111]  [<ffffffff8020a267>] ? gfn_to_hva+0x27/0x60
[ 1188.704115]  [<ffffffff8020a4a5>] ? kvm_read_guest_page+0x65/0x70
[ 1188.704119]  [<ffffffff80248f0b>] ? get_user_pages_fast+0x1db/0x2e0
[ 1188.704124]  [<ffffffff80217594>] __kvm_mmu_free_some_pages+0x24/0x40
[ 1188.704127]  [<ffffffff8021b55d>] paging64_page_fault+0x40d/0x420
[ 1188.704131]  [<ffffffff8021cfd2>] ? x86_emulate_insn+0x292/0x4d30
[ 1188.704135]  [<ffffffff8021ca75>] ? x86_decode_insn+0xc45/0xf10
[ 1188.704139]  [<ffffffff8020f98d>] ? emulate_instruction+0x1cd/0x2f0
[ 1188.704143]  [<ffffffff8021919c>] kvm_mmu_page_fault+0x1c/0x90
[ 1188.704147]  [<ffffffff80226e6f>] pf_interception+0x7f/0x190
[ 1188.704151]  [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70
[ 1188.704155]  [<ffffffff8022821a>] handle_exit+0x14a/0x2c0
[ 1188.704158]  [<ffffffff80222a3d>] ? apic_update_ppr+0x2d/0x70
[ 1188.704161]  [<ffffffff8021363f>] kvm_arch_vcpu_ioctl_run+0x62f/0xaa0
[ 1188.704165]  [<ffffffff8020b5d5>] ? kvm_vm_ioctl+0x165/0x910
[ 1188.704169]  [<ffffffff8027cde9>] ? do_futex+0x689/0x9c0
[ 1188.704174]  [<ffffffff8020cad3>] kvm_vcpu_ioctl+0x5d3/0x790
[ 1188.704177]  [<ffffffff8024eaeb>] ? __dequeue_entity+0x2b/0x50
[ 1188.704181]  [<ffffffff8024eb36>] ? set_next_entity+0x26/0x50
[ 1188.704184]  [<ffffffff802d8fa1>] vfs_ioctl+0x31/0x90
[ 1188.704189]  [<ffffffff802d92f1>] do_vfs_ioctl+0x2f1/0x4e0
[ 1188.704192]  [<ffffffff802d9562>] sys_ioctl+0x82/0xa0
[ 1188.704196]  [<ffffffff8022af6b>] system_call_fastpath+0x16/0x1b
[ 1188.704200] Code: e6 48 89 df e8 66 fe ff ff 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53
48 89 fb 48 83 ec 08 <44> 8b 4f 58 45 85 c9 0f 84 d7 00 00 00 48 8b 47 68 48 85 c0 0f
[ 1188.704226] RIP  [<ffffffff80216b4c>] mmu_page_remove_parent_pte+0xc/0x100
[ 1188.704230]  RSP <ffff8801a0dc1918>
[ 1188.704232] CR2: 0000000000000058
[ 1188.704234] ---[ end trace 35a9bc3a55fc3772 ]---
Signature · 1 line
Hans
#8 lore raw
In reply to Hans de Bruin
Quoted 21 lines · Hans de Bruin
Quoted 14 lines · Hans de Bruin
Quoted 11 lines · Marcelo Tosatti
Quoted 5 lines · Hans de Bruin

I ran memtest for 11 hours and it completed 4.7 passes with no problems. But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans,

It would be helpful if you can capture a few more KVM oopses, then.

Quoted 5 lines

v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:

Quoted 5 lines

another one. This time the vm where booted sequentialy:

[  253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003  
data 0x0
[  475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
					      ^^^

So 0x10 is 1 bit different from 0x00, which would be the no present entry for AMD.

Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.

#9 lore raw
Quoted 20 lines · Marcelo Tosatti
Quoted 15 lines · Hans de Bruin
Quoted 6 lines · Hans de Bruin
Quoted 5 lines · Marcelo Tosatti
Quoted 2 lines · Hans de Bruin

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans,

Quoted 8 lines
Quoted 2 lines

It would be helpful if you can capture a few more KVM oopses, then.

v2.6.30-rc6-144-g5805977
kvm-86-122-ge2478f5
kvm from kernel
simultaneously booting two w2k8.
One vm dies:

Quoted 4 lines

another one. This time the vm where booted sequentialy:

[  253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003  data 0x0
[  475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
Quoted 8 lines
					      ^^^

So 0x10 is 1 bit different from 0x00, which would be the no present entry for AMD.

Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.

That is still on my to do list.

Signature · 1 line
Hans
#10 lore raw
In reply to Marcelo Tosatti
Quoted 5 lines · Marcelo Tosatti
Quoted 5 lines

another one. This time the vm where booted sequentialy:

[  253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 0xc0010003  data 0x0
[  475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
Quoted 8 lines
					      ^^^

So 0x10 is 1 bit different from 0x00, which would be the no present entry for AMD.

Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.

It's actually the PCD bit, not that it affects your analysis. Strange that we see this pattern (1 bit differences on sptes) on both AMD and Intel. Very suspicious.

Signature · 1 line
error compiling committee.c: too many arguments to function
#11 lore raw
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Marcelo Tosatti
Quoted 6 lines

another one. This time the vm where booted sequentialy:

[  253.268993] kvm: 2907: cpu0 unimplemented perfctr wrmsr: 
0xc0010003  data 0x0
[  475.036542] rmap_remove: ffff8800cdb913b0 10 0->BUG
					      ^^^

So 0x10 is 1 bit different from 0x00, which would be the no present entry for AMD.

Quoted 7 lines
Quoted 3 lines

Usually an indication of hardware problems. Perhaps you want to try the test Lucas mentioned.

It's actually the PCD bit, not that it affects your analysis. Strange that we see this pattern (1 bit differences on sptes) on both AMD and Intel. Very suspicious.

On Intel there was confirmation (through memtest86) that it was memory failure. The pattern was:

0xff7ffffffffff001

#12 lore raw
In reply to Hans de Bruin
Quoted 34 lines · Hans de Bruin
Quoted 33 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 5 lines
Quoted 4 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

Hans,

The oopses below point to the possibility of a hardware problem, similar to:

https://bugzilla.redhat.com/show_bug.cgi?id=480779

Quoted 7 lines
Quoted 1 line

Can you please rule it out with memtest86?

I ran memtest for 11 hours and it completed 4.7 passes with no problems. But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Hans:

There is a memory test designed to stress disk/chipset/dma interaction. I made an implementation of it on autotest, the test module is called dma_memtest:

http://autotest.kernel.org/browser/trunk/client/tests/dma_memtest/dma_memtest.py

The work that originated this autotest implementation can be found on:

http://people.redhat.com/dledford/memtest.shtml

So you can choose either to run the autotest version or the shell script provided by Doug. Try running either of them, we might see interesting results.

Regards,

Signature · 3 lines
Lucas Meneghel Rodrigues
Software Engineer (QE)
Red Hat - Emerging Technologies
#13 lore raw
Quoted 36 lines · Lucas Meneghel Rodrigues
Quoted 27 lines · Hans de Bruin
Quoted 26 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 4 lines
Quoted 3 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

Hans,

Quoted 7 lines
Quoted 5 lines

The oopses below point to the possibility of a hardware problem, similar to:

https://bugzilla.redhat.com/show_bug.cgi?id=480779

Can you please rule it out with memtest86?

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

Quoted 7 lines

Hans:

There is a memory test designed to stress disk/chipset/dma interaction. I made an implementation of it on autotest, the test module is called dma_memtest:

http://autotest.kernel.org/browser/trunk/client/tests/dma_memtest/dma_memtest.py

[09:09:47 INFO ] Test finished after 1 iterations.
Memory test passed.
[09:09:48 DEBUG] Running 'grep MemTotal /proc/meminfo' [09:09:48 DEBUG] Running 'rpm -qa'
[09:09:49 INFO ] GOOD dma_memtest dma_memtest timestamp=1243408189 localtime=May 27 09:09:49 completed successfully
[09:09:49 DEBUG] Persistent state variable __group_level now set to 1
[09:09:49 INFO ] END GOOD dma_memtest dma_memtest timestamp=1243408189 localtime=May 27 09:09:49
[09:09:49 DEBUG] Dropping caches
[09:09:49 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'echo 3 > /proc/sys/vm/drop_caches' [09:09:52 DEBUG] Persistent state variable __group_level now set to 0
[09:09:52 INFO ] END GOOD ---- ---- timestamp=1243408192 localtime=May 27 09:09:52

Well that looks good. The web page talks about forcing the system to swap. That never happend swap usage is still 0 bytes. I installed autotest on the same lv (3 disk stripe) as the vmdisks.

Signature · 1 line
Hans
#14 lore raw
Quoted 19 lines · Hans de Bruin

[09:09:47 INFO ] Test finished after 1 iterations.
Memory test passed.
[09:09:48 DEBUG] Running 'grep MemTotal /proc/meminfo' [09:09:48 DEBUG] Running 'rpm -qa'
[09:09:49 INFO ] GOOD dma_memtest dma_memtest
timestamp=1243408189 localtime=May 27 09:09:49 completed
successfully
[09:09:49 DEBUG] Persistent state variable __group_level now set to 1
[09:09:49 INFO ] END GOOD dma_memtest dma_memtest
timestamp=1243408189 localtime=May 27 09:09:49
[09:09:49 DEBUG] Dropping caches
[09:09:49 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'sync'
[09:09:51 DEBUG] Running 'echo 3 > /proc/sys/vm/drop_caches' [09:09:52 DEBUG] Persistent state variable __group_level now set to 0
[09:09:52 INFO ] END GOOD ---- ---- timestamp=1243408192
localtime=May 27 09:09:52

Quoted 3 lines

Well that looks good. The web page talks about forcing the system to swap. That never happend swap usage is still 0 bytes. I installed autotest on the same lv (3 disk stripe) as the vmdisks.

Interesting, about that I made some tests and just realized that in some cases both mine and the original shell implementation are failing on forcing the system to go to swap (the tests I made when the test was firstly implemented did manage to make the machines swap, but in conditions where the system had a significantly higher initial memory usage). I will work on a better heuristic to force swap.

Thanks for pointing this out,

Signature · 3 lines
Lucas Meneghel Rodrigues
Software Engineer (QE)
Red Hat - Emerging Technologies
#15 lore raw
In reply to Hans de Bruin
Quoted 29 lines · Hans de Bruin
Quoted 28 lines · Marcelo Tosatti
Quoted 19 lines · Hans de Bruin
Quoted 18 lines · Hans de Bruin

Staring to vms simultaneously end in crash

linux 30-rc5
kvm-qemu kvm-85-378-g143eb2b
proc AMD dualcore

vm's like:

#!/bin/sh
n=10
cdrom=/iso/server2008x64.iso
drive=file=/kvm/disks/vm$n
mem=1024
cpu=qemu64
vga=std
mac=52:54:00:12:34:$n
bridge=br1

Quoted 5 lines
Quoted 4 lines

qemu-system-x86_64 -cdrom $cdrom -drive $drive -m $mem -cpu $cpu -vga $vga -net nic,macaddr=$mac -net tap,script=/etc/qemu/$bridge

another dmesg:

Hans,

Quoted 9 lines
Quoted 5 lines

The oopses below point to the possibility of a hardware problem, similar to:

https://bugzilla.redhat.com/show_bug.cgi?id=480779

Can you please rule it out with memtest86?

I ran memtest for 11 hours and it completed 4.7 passes with no problems.
But then memtest is about cpu and memmory interaction. If the problem is disk related there is also disk/chipset/dma and memmory interaction. I could degrade my system by turning dma on disk io off, or i could have a closer look at kvm-autotest.

And now summer has arived, and the temperatures are going up. Instead of just the kvm troubles, its now impossible to compile a kernel. I am going to replace my hardware.

Signature · 1 line
Hans
#16 lore raw
Quoted 1 line · Hans de Bruin

And now summer has arived, and the temperatures are going up. Instead of just the kvm troubles, its now impossible to compile a kernel. I am going to replace my hardware.

Apparently the virtualization extensions stress hardware. Maybe it's the long microcode sequences.

Signature · 1 line
error compiling committee.c: too many arguments to function