how to use mkfs.ext3 "stride=" on LVM on RAID correctly?
Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem placed on LVM, which was placed on Linux RAID-5.
Luckily, it was possible to alleviate much of this problem by removing internal bitmap from RAID-5 and by using anticipatory IO scheduler.
As suggested earlier by Theodore Tso in "very poor ext3 write performance on big filesystems" thread on linux-fsdevel list, I should have used "mke2fs -E stride option to make sure all of the bitmaps don't get concentrated on one hard drive spindle".
Although the fine mkfs.ext3 manual gives some basic information on how to do it if you place the whole filesystem on a RAID array, it is not clear to me how it should be done correctly if I want to create a ext3 filesystem on LVM on RAID-5.
Any helpful hints?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 13 lines · Tomasz Chmielewski
Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem
placed on LVM, which was placed on Linux RAID-5.Luckily, it was possible to alleviate much of this problem by removing internal bitmap from RAID-5 and by using anticipatory IO scheduler.
As suggested earlier by Theodore Tso in "very poor ext3 write performance on big filesystems" thread on linux-fsdevel list, I should have used "mke2fs -E stride option to make sure all of the bitmaps don't get concentrated on one hard drive spindle".
Quoted 6 lines
Although the fine mkfs.ext3 manual gives some basic information on how to do it if you place the whole filesystem on a RAID array, it is not clear to me how it should be done correctly if I want to create a ext3 filesystem on LVM on RAID-5.
Any helpful hints?
I can't help with the stride stuff, as I use LVM and that makes the stride stuff utterly useless anyway. However, a suggestion I have for you which I found helped a great deal was to place the bitmap on some /other/ device's filesystem (I used a /etc/bitmaps/). Obviously, don't put it on a FS which is part of the raid, I typically divide each device into 2 or more raids, but only if I don't have enough devices for /system/ and /data/ -style pools.
Signature · 1 line
Jon
Quoted 3 lines · Tomasz Chmielewski
Although the fine mkfs.ext3 manual gives some basic information on how to do it if you place the whole filesystem on a RAID array, it is not clear to me how it should be done correctly if I want to create a ext3 filesystem on LVM on RAID-5.
Any helpful hints?
You should be careful with all the alignemnts. Extents should be properly aligned with - at the very least - stripes, and preferably with whole stripe width (not always possible though). To check where lvm extents, you can use i.e.:
dmsetup table
pvs -o+pe_start --units s
losetup and hexedit experiments (silly, but I used to do it like that in the past, before I realised about other options :o )
If you don't need default 4M granularity, it's nice to increase it (vgcreate -s option ; I use 256M and 512M sized extents). You can alter the beginning of the extents with --metadatasize option of pvcreate (overally it always rounds up to the next 64K multiple). Be sure to verify with one of the commands above.
Moreover - if you use partitionable raid and create lvm in one of the partitions, make sure partition is properly aligned as well.