KVM and kernel 2.6.30 file system madness

Hi there,

back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to crash from time to time. That time we used XFS as filesystem (/ and /boot where ext3/ext2). Since XFS worked so very well for us on physical hosts the natural choise for our OSs in KVM of course was also XFS. This was a bad idea because it caused some filesystem corruptions on some KVMs when KVM crashed (without any message).
Somewhere I read that XFS in KVM should only be used with the KVM parameter "cache=none". Since then this is now our default for all KVMs (even with ext3). I thought by myself that KVM and an FS which does heavy write caching like XFS is a bad choise so I decided that I can't trust XFS inside a KVM anymore and so I switched all filesystems in our KVMs to ext3. This was a good choice. No FS corruptions anymore - well and no unplaned crashes of of KVM too ;-)
Since yesterday (no crash but FS corruptions)...

I installed kernel 2.6.30-r2 in one of our guests. This was a not so good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5 and KVM is 84 (KVM 85 has issues with VNC display and 86 and 87 not in portage currently). Using qow2 as KVM image format.

I installed all the stuff we needed in the new KVM and a Postgres database. But something was different. The database import was suddenly fast as hell. I've never seen such good I/O throughput in a KVM. Well after almost finished with the whole installation process I noticed some strange ext3 messages in the "dmesg" output. "Oh no... Not again problems with FS corruptions" I thought... Well after a reboot of the KVM it was sure that the rootfs was corrupted. /etc/hostname and some other files suddenly were binary files :-( Lukely I was able to correct the problems with fsck and get the files back from the backup.

So what happend in 2.6.30? Ah... I remembered immediately that the kernel developers decided to switch the default value of the journaling mode (data=...) from "ordered" to "writeback". Well... Now I know why the database import was so fast... But at what price? I'm really curious what happens when the major distributions roll out their distributions with this default option.

So my question is: I'm the only one in the universe with this FS problems? Am I completely wrong here? Is "data=ordered" the recommended mode for ext3 in KVMs and even necessary when KVM ist not crashing? This kind of stuff sometimes makes live to so easy... ;-)

Are you using virtio-block?

In any case, not using a released version always has risks.

Quoted 33 lines · Robert Wimmer

Hi there,

back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to crash from time to time. That time we used XFS as filesystem (/ and /boot where ext3/ext2). Since XFS worked so very well for us on physical hosts the natural choise for our OSs in KVM of course was also XFS. This was a bad idea because it caused some filesystem corruptions on some KVMs when KVM crashed (without any message).
Somewhere I read that XFS in KVM should only be used with the KVM parameter "cache=none". Since then this is now our default for all KVMs (even with ext3). I thought by myself that KVM and an FS which does heavy write caching like XFS is a bad choise so I decided that I can't trust XFS inside a KVM anymore and so I switched all filesystems in our KVMs to ext3. This was a good choice. No FS corruptions anymore - well and no unplaned crashes of of KVM too ;-)
Since yesterday (no crash but FS corruptions)...

I installed kernel 2.6.30-r2 in one of our guests. This was a not so good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5 and KVM is 84 (KVM 85 has issues with VNC display and 86 and 87 not in portage currently). Using qow2 as KVM image format.

I installed all the stuff we needed in the new KVM and a Postgres database. But something was different. The database import was suddenly fast as hell. I've never seen such good I/O throughput in a KVM. Well after almost finished with the whole installation process I noticed some strange ext3 messages in the "dmesg" output. "Oh no... Not again problems with FS corruptions" I thought... Well after a reboot of the KVM it was sure that the rootfs was corrupted. /etc/hostname and some other files suddenly were binary files :-( Lukely I was able to correct the problems with fsck and get the files back from the backup.

Quoted 7 lines

So what happend in 2.6.30? Ah... I remembered immediately that the kernel developers decided to switch the default value of the journaling mode (data=...) from "ordered" to "writeback". Well... Now I know why the database import was so fast... But at what price? I'm really curious what happens when the major distributions roll out their distributions with this default option.

Distributions will likely change the default.

Quoted 6 lines

So my question is: I'm the only one in the universe with this FS problems? Am I completely wrong here? Is "data=ordered" the recommended mode for ext3 in KVMs and even necessary when KVM ist not crashing? This kind of stuff sometimes makes live to so easy... ;-)

Are you using virtio-block?

In any case, not using a released version always has risks.

Amit

Hi!

Quoted 1 line

Are you using virtio-block?

Yes.

Quoted 1 line

In any case, not using a released version always has risks.

Well, what do you mean by "not using a released version"? The package "gentoo-sources" always uses released kernels. 2.6.30-r2 in Gentoo means that this is the third update of the stable kernel version 2.6.30. The "-r*" releases just contains small patches from the Gentoo kernel package maintainers.

Quoted 5 lines · Robert Wimmer

Hi!

Quoted 1 line

Are you using virtio-block?

Yes.

OK, then there is a known problem. I think the fix is waiting to be applied.

Quoted 3 lines · Amit Shah

Quoted 2 lines · Robert Wimmer

Hi!

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

Are you using virtio-block?

Yes.

OK, then there is a known problem. I think the fix is waiting to be applied.

Amit, would you kindly state the problem with virtio-block?