both qemu-kvm-0.10.6 and kvm-88 crashing under heavy load while using scsi-backed storage
Hello,
we're having problem one of our kvm guests. According to storage summary mail Christopher sent few days ago, I switched my disk model to SCSI which should be the safest choice. But now we've stumbled upon the problem that the whole guest crashes when we run one specific postgres query which heavily loads it.
I can 100% reproduce this problem on both our production nodes (each 8cores, 16GB RAM), on my testing machine (4cores, 3GB RAM) this causes the HOST to reboot (which is even worse). We haven't experienced this problem with virtio.
I tried both qemu-kvm-0.10.6 and kvm-88, the host is running 2.6.30.5, I tried 2.6.29.x and 2.6.30.5 for guest.
Guest backtrace follows (it's a bit mangled as it's obtained using netconsole)
Sep 2 19:01:20 sql2 [ 1564.795629] BUG: unable to handle kernel Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.798969] PGD 2d5a2067 Sep 2 19:01:20 sql2 PUD 2d4ad067 Sep 2 19:01:20 sql2 [ 1564.798969] PGD 2d5a2067 Sep 2 19:01:20 sql2 PMD 0 Sep 2 19:01:20 sql2 PUD 2d4ad067 Sep 2 19:01:20 sql2 PMD 0 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.799505] Oops: 0000 [#1] Sep 2 19:01:20 sql2 PREEMPT Sep 2 19:01:20 sql2 [ 1564.799505] Oops: 0000 [#1] Sep 2 19:01:20 sql2 SMP Sep 2 19:01:20 sql2 PREEMPT Sep 2 19:01:20 sql2 SMP Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] last sysfs file: /sys/block/md0/dev Sep 2 19:01:20 sql2 [ 1564.800773] CPU 3 Sep 2 19:01:20 sql2 [ 1564.800773] last sysfs file: /sys/block/md0/dev Sep 2 19:01:20 sql2 [ 1564.800773] CPU 3 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Modules linked in: Sep 2 19:01:20 sql2 netconsole Sep 2 19:01:20 sql2 [ 1564.800773] Modules linked in: Sep 2 19:01:20 sql2 ipv6 Sep 2 19:01:20 sql2 netconsole Sep 2 19:01:20 sql2 ipv6 Sep 2 19:01:20 sql2 reiserfs Sep 2 19:01:20 sql2 dm_mirror Sep 2 19:01:20 sql2 reiserfs Sep 2 19:01:20 sql2 dm_mirror Sep 2 19:01:20 sql2 dm_region_hash Sep 2 19:01:20 sql2 dm_log Sep 2 19:01:20 sql2 dm_region_hash Sep 2 19:01:20 sql2 dm_log Sep 2 19:01:20 sql2 dm_mod Sep 2 19:01:20 sql2 video Sep 2 19:01:20 sql2 dm_mod Sep 2 19:01:20 sql2 video Sep 2 19:01:20 sql2 backlight Sep 2 19:01:20 sql2 output Sep 2 19:01:20 sql2 backlight Sep 2 19:01:20 sql2 output Sep 2 19:01:20 sql2 sbs Sep 2 19:01:20 sql2 sbshc Sep 2 19:01:20 sql2 sbs Sep 2 19:01:20 sql2 sbshc Sep 2 19:01:20 sql2 fan Sep 2 19:01:20 sql2 container Sep 2 19:01:20 sql2 fan Sep 2 19:01:20 sql2 battery Sep 2 19:01:20 sql2 container Sep 2 19:01:20 sql2 battery Sep 2 19:01:20 sql2 ac Sep 2 19:01:20 sql2 ac Sep 2 19:01:20 sql2 parport_pc Sep 2 19:01:20 sql2 lp Sep 2 19:01:20 sql2 parport_pc Sep 2 19:01:20 sql2 lp Sep 2 19:01:20 sql2 parport Sep 2 19:01:20 sql2 nvram Sep 2 19:01:20 sql2 parport Sep 2 19:01:20 sql2 nvram Sep 2 19:01:20 sql2 sg Sep 2 19:01:20 sql2 sr_mod Sep 2 19:01:20 sql2 sg Sep 2 19:01:20 sql2 sr_mod Sep 2 19:01:20 sql2 cdrom Sep 2 19:01:20 sql2 thermal Sep 2 19:01:20 sql2 cdrom Sep 2 19:01:20 sql2 thermal Sep 2 19:01:20 sql2 processor Sep 2 19:01:20 sql2 thermal_sys Sep 2 19:01:20 sql2 processor Sep 2 19:01:20 sql2 button Sep 2 19:01:20 sql2 thermal_sys Sep 2 19:01:20 sql2 button Sep 2 19:01:20 sql2 virtio_balloon Sep 2 19:01:20 sql2 virtio_balloon Sep 2 19:01:20 sql2 i2c_piix4 Sep 2 19:01:20 sql2 e1000 Sep 2 19:01:20 sql2 i2c_piix4 Sep 2 19:01:20 sql2 i2c_core Sep 2 19:01:20 sql2 e1000 Sep 2 19:01:20 sql2 piix Sep 2 19:01:20 sql2 i2c_core Sep 2 19:01:20 sql2 piix Sep 2 19:01:20 sql2 sym53c8xx Sep 2 19:01:20 sql2 scsi_transport_spi Sep 2 19:01:20 sql2 sym53c8xx Sep 2 19:01:20 sql2 pcspkr Sep 2 19:01:20 sql2 scsi_transport_spi Sep 2 19:01:20 sql2 pcspkr Sep 2 19:01:20 sql2 virtio_blk Sep 2 19:01:20 sql2 virtio_net Sep 2 19:01:20 sql2 virtio_blk Sep 2 19:01:20 sql2 virtio_pci Sep 2 19:01:20 sql2 virtio_net Sep 2 19:01:20 sql2 virtio_ring Sep 2 19:01:20 sql2 virtio_pci Sep 2 19:01:20 sql2 virtio Sep 2 19:01:20 sql2 virtio_ring Sep 2 19:01:20 sql2 8139cp Sep 2 19:01:20 sql2 virtio Sep 2 19:01:20 sql2 8139cp Sep 2 19:01:20 sql2 mii Sep 2 19:01:20 sql2 bitrev Sep 2 19:01:20 sql2 mii Sep 2 19:01:20 sql2 bitrev Sep 2 19:01:20 sql2 pata_acpi Sep 2 19:01:20 sql2 pata_acpi Sep 2 19:01:20 sql2 ide_pci_generic Sep 2 19:01:20 sql2 ide_core Sep 2 19:01:20 sql2 ide_pci_generic Sep 2 19:01:20 sql2 ata_piix Sep 2 19:01:20 sql2 ide_core Sep 2 19:01:20 sql2 ata_generic Sep 2 19:01:20 sql2 ata_piix Sep 2 19:01:20 sql2 libata Sep 2 19:01:20 sql2 ata_generic Sep 2 19:01:20 sql2 sd_mod Sep 2 19:01:20 sql2 libata Sep 2 19:01:20 sql2 scsi_mod Sep 2 19:01:20 sql2 sd_mod Sep 2 19:01:20 sql2 ext4 Sep 2 19:01:20 sql2 scsi_mod Sep 2 19:01:20 sql2 jbd2 Sep 2 19:01:20 sql2 ext4 Sep 2 19:01:20 sql2 jbd2 Sep 2 19:01:20 sql2 crc32 Sep 2 19:01:20 sql2 crc16 Sep 2 19:01:20 sql2 crc32 Sep 2 19:01:20 sql2 uhci_hcd Sep 2 19:01:20 sql2 crc16 Sep 2 19:01:20 sql2 ohci_hcd Sep 2 19:01:20 sql2 uhci_hcd Sep 2 19:01:20 sql2 ehci_hcd Sep 2 19:01:20 sql2 ohci_hcd Sep 2 19:01:20 sql2 ehci_hcd Sep 2 19:01:20 sql2 [last unloaded: scsi_wait_scan] Sep 2 19:01:20 sql2 [last unloaded: scsi_wait_scan] Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Pid: 0, comm: swapper Not tainted 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.800773] Pid: 0, comm: swapper Not tainted 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.800773] RIP: 0010:[<ffffffffa018beff>] Sep 2 19:01:20 sql2 [ 1564.800773] RIP: 0010:[<ffffffffa018beff>] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] RSP: 0018:ffff88000244edf8 EFLAGS: 00010087 Sep 2 19:01:20 sql2 [ 1564.800773] RSP: 0018:ffff88000244edf8 EFLAGS: 00010087 Sep 2 19:01:20 sql2 [ 1564.800773] RAX: 000000000000000b RBX: 000000000000000b RCX: 0000000000000084 Sep 2 19:01:20 sql2 [ 1564.800773] RAX: 000000000000000b RBX: 000000000000000b RCX: 0000000000000084 Sep 2 19:01:20 sql2 [ 1564.800773] RDX: ffff88002e8dc800 RSI: 000000002d46b168 RDI: ffffc2000021a006 Sep 2 19:01:20 sql2 [ 1564.800773] RDX: ffff88002e8dc800 RSI: 000000002d46b168 RDI: ffffc2000021a006 Sep 2 19:01:20 sql2 [ 1564.800773] RBP: ffff88000244ee68 R08: ffffffffa00b243c R09: 00017fd30f1b243c Sep 2 19:01:20 sql2 [ 1564.800773] RBP: ffff88000244ee68 R08: ffffffffa00b243c R09: 00017fd30f1b243c Sep 2 19:01:20 sql2 [ 1564.800773] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002d46b168 Sep 2 19:01:20 sql2 [ 1564.800773] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002d46b168 Sep 2 19:01:20 sql2 [ 1564.800773] R13: 0000000000000000 R14: ffff88002d46b000 R15: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] R13: 0000000000000000 R14: ffff88002d46b000 R15: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] FS: 0000000000000000(0000) GS:ffff88000244b000(0000) knlGS:0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] FS: 0000000000000000(0000) GS:ffff88000244b000(0000) knlGS:0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 2 19:01:20 sql2 [ 1564.800773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 2 19:01:20 sql2 [ 1564.800773] CR2: 0000000000000358 CR3: 000000002dd34000 CR4: 00000000000006e0 Sep 2 19:01:20 sql2 [ 1564.800773] CR2: 0000000000000358 CR3: 000000002dd34000 CR4: 00000000000006e0 Sep 2 19:01:20 sql2 [ 1564.800773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 2 19:01:20 sql2 [ 1564.800773] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 2 19:01:20 sql2 [ 1564.800773] Process swapper (pid: 0, threadinfo ffff88002f8c2000, task ffff88002f8c8000) Sep 2 19:01:20 sql2 [ 1564.800773] Process swapper (pid: 0, threadinfo ffff88002f8c2000, task ffff88002f8c8000) Sep 2 19:01:20 sql2 [ 1564.800773] Stack: Sep 2 19:01:20 sql2 [ 1564.800773] 0000000000000040 Sep 2 19:01:20 sql2 ffff88000244eec0 Sep 2 19:01:20 sql2 [ 1564.800773] 0000000000000040 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 ffff88000244eec0 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 ffff88000245bfc0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 0000000000000046 Sep 2 19:01:20 sql2 ffff88000245bfc0 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Call Trace: Sep 2 19:01:20 sql2 [ 1564.800773] <IRQ> Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [<ffffffffa018e11c>] sym_interrupt+0x43c/0x720 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffffa0186b61>] sym53c8xx_intr+0x41/0x90 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8028ecac>] handle_IRQ_event+0x4c/0x1c0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff80290eb9>] handle_fasteoi_irq+0x89/0x110 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020e05f>] handle_irq+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020d81e>] do_IRQ+0x6e/0x100 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020bd53>] ret_from_intr+0x0/0xa Sep 2 19:01:20 sql2 [ 1564.800773] <EOI> Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [<ffffffff802130c5>] ? default_idle+0x45/0x150 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8025da81>] ? atomic_notifier_call_chain+0x11/0x20 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020a20e>] ? cpu_idle+0x4e/0xa0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff804e5aa3>] ? start_secondary+0x173/0x1d0 Sep 2 19:01:20 sql2 [ 1564.800773] Code: Sep 2 19:01:20 sql2 ad Sep 2 19:01:20 sql2 07 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 2c Sep 2 19:01:20 sql2 01 Sep 2 19:01:20 sql2 0f Sep 2 19:01:20 sql2 84 Sep 2 19:01:20 sql2 5d Sep 2 19:01:20 sql2 09 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 0f Sep 2 19:01:20 sql2 ae Sep 2 19:01:20 sql2 f8 Sep 2 19:01:20 sql2 49 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 b6 Sep 2 19:01:20 sql2 50 Sep 2 19:01:20 sql2 05 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 41 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 be Sep 2 19:01:20 sql2 60 Sep 2 19:01:20 sql2 06 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 48 Sep 2 19:01:20 sql2 83 Sep 2 19:01:20 sql2 c6 Sep 2 19:01:20 sql2 2c Sep 2 19:01:20 sql2 e8 Sep 2 19:01:20 sql2 e6 Sep 2 19:01:20 sql2 64 Sep 2 19:01:20 sql2 1f Sep 2 19:01:20 sql2 e0 Sep 2 19:01:20 sql2 e9 Sep 2 19:01:20 sql2 4d Sep 2 19:01:20 sql2 fe Sep 2 19:01:20 sql2 ff Sep 2 19:01:20 sql2 ff Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 97 Sep 2 19:01:20 sql2 58 Sep 2 19:01:20 sql2 03 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 48 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 82 Sep 2 19:01:20 sql2 80 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 4c Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 a0 Sep 2 19:01:20 sql2 c8 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.878205] RIP Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] RSP <ffff88000244edf8> Sep 2 19:01:20 sql2 [ 1564.878205] CR2: 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.878205] ---[ end trace 1af2f1cc92726b74 ]--- Sep 2 19:01:20 sql2 [ 1564.878205] Kernel panic - not syncing: Fatal exception in interrupt Sep 2 19:01:20 sql2 [ 1564.878205] Pid: 0, comm: swapper Tainted: G D 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.878205] Call Trace: Sep 2 19:01:20 sql2 [ 1564.878205] <IRQ> Sep 2 19:01:20 sql2 [<ffffffff80240cda>] panic+0xaa/0x180 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff804ecee0>] ? _spin_unlock_irqrestore+0x50/0x60 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8024146f>] ? release_console_sem+0x1df/0x200 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80241707>] ? console_unblank+0x67/0x80 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8024095e>] ? print_oops_end_marker+0x1e/0x20 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020f74f>] oops_end+0x9f/0xb0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8022871a>] no_context+0x15a/0x250 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80232730>] ? __enqueue_entity+0x80/0x90 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff802289ab>] __bad_area_nosemaphore+0xdb/0x1c0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80234499>] ? find_busiest_group+0x1a9/0x940 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80228b2e>] bad_area_nosemaphore+0xe/0x10 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80228e4b>] do_page_fault+0xab/0x270 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff804ed36f>] page_fault+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018beff>] ? sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018bc9e>] ? sym_int_sir+0x5e/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018e11c>] sym_interrupt+0x43c/0x720 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa0186b61>] sym53c8xx_intr+0x41/0x90 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8028ecac>] handle_IRQ_event+0x4c/0x1c0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80290eb9>] handle_fasteoi_irq+0x89/0x110 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020e05f>] handle_irq+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020d81e>] do_IRQ+0x6e/0x100 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020bd53>] ret_from_intr+0x0/0xa Sep 2 19:01:20 sql2 [ 1564.878205] <EOI> Sep 2 19:01:20 sql2 [<ffffffff802130c5>] ? default_idle+0x45/0x150 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8025da81>] ? atomic_notifier_call_chain+0x11/0x20 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020a20e>] ? cpu_idle+0x4e/0xa0
Any idea on what could be wrong?
I'll gladly provide any further information which might help to track this bug down... thanks a lot
regards
nik
Signature · 14 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 6 lines · Nikola Ciprich
Hello,
we're having problem one of our kvm guests. According to storage summary mail Christopher sent few days ago, I switched my disk model to SCSI which should be the safest choice. But now we've stumbled upon the problem that the whole guest crashes when we run one specific postgres query which heavily loads it.
scsi is hardly tested, so this isn't surprising.
Quoted 4 lines
I can 100% reproduce this problem on both our production nodes (each 8cores, 16GB RAM), on my testing machine (4cores, 3GB RAM) this causes the HOST to reboot (which is even worse). We haven't experienced this problem with virtio.
AMD or Intel? Uni or smp guests?
The host crash is of course more worrying. Can you capture dmesg?
Quoted 4 lines
I tried both qemu-kvm-0.10.6 and kvm-88, the host is running 2.6.30.5, I tried 2.6.29.x and 2.6.30.5 for guest.
Guest backtrace follows (it's a bit mangled as it's obtained using netconsole)
Quoted 10 lines
Sep 2 19:01:20 sql2 [ 1564.795629] BUG: unable to handle kernel Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx]
It's in the scsi driver, so it's probable our scsi emulation is broken. Or maybe (unlikely) a bug in the driver.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Hello Avi, thnaks for reply.
Quoted 1 line
scsi is hardly tested, so this isn't surprising.
well, but if it's still the most safe way to go, then there's not much other choices, is it?
Quoted 1 line
AMD or Intel? Uni or smp guests?
intel, SMP guests. sorry for not reporting this right away..
Quoted 1 line
The host crash is of course more worrying. Can you capture dmesg?
there's absolutely no output, just immediate reboot :(
Quoted 3 lines
It's in the scsi driver, so it's probable our scsi emulation is broken. Or maybe (unlikely) a bug in the driver.
Well, I guess it'll be in emulation as well, as the driver is quite old so it should be well tested.
Any hints on how could I futher debug it?
thanks a lot once again
nik
Signature · 14 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 9 lines · Nikola Ciprich
Hello Avi, thnaks for reply.
Quoted 2 lines
scsi is hardly tested, so this isn't surprising.
well, but if it's still the most safe way to go, then there's not much other choices, is it?
If it crashes, it isn't the safest way.
Quoted 11 lines
Quoted 2 lines
AMD or Intel? Uni or smp guests?
intel, SMP guests. sorry for not reporting this right away..
Quoted 2 lines
The host crash is of course more worrying. Can you capture dmesg?
there's absolutely no output, just immediate reboot :(
Not even through netconsole? It's on just one host, right?
Quoted 8 lines
Quoted 3 lines
It's in the scsi driver, so it's probable our scsi emulation is broken. Or maybe (unlikely) a bug in the driver.
Well, I guess it'll be in emulation as well, as the driver is quite old so it should be well tested.
Any hints on how could I futher debug it?
Add printk()s in the driver and see why it's confused. Maybe it will tell us something about the bug in the device.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
I tried running the test on my test machine with 2.6.31-rc8 guest kernel and I wasn't able to reproduce the problem...
So it's possible that the problem could have been fixed somewhere between 2.6.30..2.6.31-rc8
Do You have any idea on what could be the related fix? It would be a good candidate for 2.6.30.6 stable release...
I'll proceed with further testing...
Signature · 13 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 9 lines · Nikola Ciprich
I tried running the test on my test machine with 2.6.31-rc8 guest kernel and I wasn't able to reproduce the problem...
So it's possible that the problem could have been fixed somewhere between 2.6.30..2.6.31-rc8
Do You have any idea on what could be the related fix? It would be a good candidate for 2.6.30.6 stable release...
I'll proceed with further testing...
A bisect could help, though it would be a long process. Running
git bisect arch/x86/kvm virt/kvm
will speed it up, expect around 8 tests and 2-3 full recompiles.
Signature · 1 line
error compiling committee.c: too many arguments to function
Yup,
I was lazily trying to avoid that :)
OK, I'll give it a try and report..
Quoted 25 lines · Avi Kivity
Quoted 9 lines · Nikola Ciprich
I tried running the test on my test machine with 2.6.31-rc8 guest kernel and I wasn't able to reproduce the problem...
So it's possible that the problem could have been fixed somewhere between 2.6.30..2.6.31-rc8
Do You have any idea on what could be the related fix? It would be a good candidate for 2.6.30.6 stable release...
I'll proceed with further testing...A bisect could help, though it would be a long process. Running
git bisect
arch/x86/kvmvirt/kvmwill speed it up, expect around 8 tests and 2-3 full recompiles.
Signature · 6 lines
error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 13 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------