DMA errors in guest caused by corrupted(?) disk image
Hi,
I've just come across a somewhat strange problem that was suggested I report to the list.
The problem manifested itself as DMA errors and the like popping up in the guest, like I'd expect to see if a disk in a physical machine was dying, like this:
hda: dma_timer_expiry: dma status == 0x21
The VM has previously been quite stable until this problem started part of the way through today. Another guest on the same host machine is fine.
After talking to iggy on IRC, I tried running qemu-convert over the disk image to copy it to another image, and that solved the problem. So it looks like disk image corruption somehow manages to manifest itself as a DMA error in the guest...
I'm starting KVM like so:
kvm -m 512 -net nic,macaddr=$macaddr -net tap,iface=$iface -hda hda.qc2
As the filename suggests, it's a qcow2 image, 30GB in size.
The guest is a 32 bit RHEL3 installation. The host is a 64 bit Debian Lenny machine, running kvm 84.
I've got a copy of the dodgy disk image, although it's 2.7GB so not so easy to ship around. I can do any diagnostics on it that people need to try and track down the cause of the problem. I tried doing another qemu-convert (with a view to seeing the differences between the two images) but the copy is 500MB smaller (zero blocks removed, presumably) so a diff probably isn't going to help much.
- Matt
Quoted 11 lines · Matthew Palmer appeared and
Hi,
I've just come across a somewhat strange problem that was suggested I report to the list.
The problem manifested itself as DMA errors and the like popping up in the guest, like I'd expect to see if a disk in a physical machine was dying, like this:
hda:
dma_timer_expiry: dma status ==0x21
[...]
I've seen a similar effect on my KVM host server. There were I/O errors on one partition. When I tried to reformat the partition and restore its content from backups, mkfs was reporting I/O errors, always at the same block number. I added a second qcow2 file and moved the partition there.
Quoted 2 lines
The VM has previously been quite stable until this problem started part of the way through today. Another guest on the same host machine is fine.
The KVM host runs since January very stable. The system is a Debian Lenny with custom kernel 2.6.28.8 and kvm-84. The filesystem for images is ext4 and the guests use ext4 in part, too.
I cannot provide the images since they are quite large (about 20 to 44 GB) and they contain sensitive data.
I plan to move to 2.6.29.x or 2.6.30.x for reasons of the ext4 patches regarding delayed allocation and to upgrade kvm, but I can to tests with the current system if you wish.
Best,
René.
Signature · 4 lines
)\._.,--....,'``. fL Let GNU/Linux work for you while you take a nap. /, _.. \ _\ (`._ ,. R. Pfeiffer <lynx at luchs.at> + http://web.luchs.at/ `._.-(,_..'--(,_..'`-.;.' - System administration + Consulting + Teaching - Got mail delivery problems? http://web.luchs.at/information/blockedmail.php
Quoted 11 lines · René Pfeiffer
Quoted 9 lines · Matthew Palmer appeared and
Hi,
I've just come across a somewhat strange problem that was suggested I report to the list.
The problem manifested itself as DMA errors and the like popping up in the guest, like I'd expect to see if a disk in a physical machine was dying, like this:
Quoted 10 lines
Quoted 3 lines
hda:
dma_timer_expiry: dma status ==0x21
[...]I've seen a similar effect on my KVM host server. There were I/O errors on one partition. When I tried to reformat the partition and restore its content from backups, mkfs was reporting I/O errors, always at the same block number. I added a second qcow2 file and moved the partition there.
It's worth trying a qemu-img convert to move the image to an LVM volume.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.