XFS over LVM over md RAID
Using the latest, stable versions of LVM2 and xfsprogs and the 2.6.35.4 kernel, I am setting up lvm on a 16 drive, 256k chunk md RAID6, which has been used to date with XFS directly on the RAID.
mkfs.xfs directly on the RAID gives:
meta-data=/dev/md8 isize=256 agcount=32, agsize=106814656 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=3418068864, imaxpct=5
= sunit=64 swidth=896 blks
naming =version 2 bsize=4096 ascii-ci=0which gives the correct sunit and swidth values for the array.
Creating an lv which uses the entire array and mkfs.xfs on that, gives:
meta-data=/dev/vg_local/Storage isize=256 agcount=13, agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=3418067968, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0Limited testing using dd and bonnie++ shows no difference in write performance whether I use sunit=64/swidth=896 or sunit=0/swidth=0 on the lv.
My gut reaction is that I should be using 64/896 but maybe mkfs.xfs knows better?
Regards,
Richard
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · On Freitag, 10. September 2010 Richard Scobie wrote:
Limited testing using dd and bonnie++ shows no difference in write performance
For dd it should always show the same, as you just sequentially write a large file. Only with bonnie you would see differences due to stripe sets, as the speed only drops when doing I/O not on boundaries and/or smaller than the stripe size, as the stripe size is the smallest possible I/O for the RAID.
I don't know why you don't see any difference with bonnie though.
FWIW, a stripe set of 256k means you do read/write 256k from a single drive on each I/O, then the next 256k from the next drive. I hope you have very few small accesses and mostly very large files. If you'd use a database on that system it would crawl...
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Michael Monnerie
FWIW, a stripe set of 256k means you do read/write 256k from a single drive on each I/O, then the next 256k from the next drive. I hope you have very few small accesses and mostly very large files. If you'd use a database on that system it would crawl...
Mostly > 2MB and multi GB.
Regards,
Richard
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · Michael Monnerie
I don't know why you don't see any difference with bonnie though.
Re-ran the bonnie tests using 1.2MB files instead of the 12MB files i used initially and there is a substantial difference - sorry for the noise.
Regards,
Richard
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Richard Scobie
Using the latest, stable versions of LVM2 and xfsprogs and the 2.6.35.4 kernel, I am setting up lvm on a 16 drive, 256k chunk md RAID6, which has been used to date with XFS directly on the RAID.
mkfs.xfs directly on the RAID gives:
meta-data=/dev/md8 isize=256 agcount=32,
agsize=106814656 blks= sectsz=4096 attr=2 data = bsize=4096 blocks=3418068864, imaxpct=5 = sunit=64 swidth=896 blks naming =version 2 bsize=4096 ascii-ci=0which gives the correct sunit and swidth values for the array.
Quoted 8 lines
Creating an lv which uses the entire array and mkfs.xfs on that, gives:
meta-data=/dev/vg_local/Storage isize=256 agcount=13,
agsize=268435455 blks= sectsz=512 attr=2 data = bsize=4096 blocks=3418067968, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0
Hmmm - it's treating MD very differently to the LVM volume - different numbers of AGs, different sunit/swdith. Did you build xfsprogs yourself? Is it linked against libblkid or libdisk?
Or it might be that LVM is not exporting the characteristic of the underlying volume. Can you check if there are different parameter values exported by the two devices in /sys/block/<dev>/queue?
Quoted 3 lines
Limited testing using dd and bonnie++ shows no difference in write performance whether I use sunit=64/swidth=896 or sunit=0/swidth=0 on the lv.
These benchmarks won't realy show any difference on an empty filesystem. It will have an impact on how the filesystems age and how well aligned the IO will be to the underlying device under more complex workloads...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave,
Quoted 25 lines · Dave Chinner
Quoted 15 lines · Richard Scobie
Using the latest, stable versions of LVM2 and xfsprogs and the 2.6.35.4 kernel, I am setting up lvm on a 16 drive, 256k chunk md RAID6, which has been used to date with XFS directly on the RAID.
mkfs.xfs directly on the RAID gives:
meta-data=/dev/md8 isize=256 agcount=32,
agsize=106814656 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=3418068864, imaxpct=5
= sunit=64 swidth=896 blks
naming =version 2 bsize=4096 ascii-ci=0which gives the correct sunit and swidth values for the array.
Quoted 8 lines
Creating an lv which uses the entire array and mkfs.xfs on that, gives:
meta-data=/dev/vg_local/Storage isize=256 agcount=13,
agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=3418067968, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
Quoted 3 lines
Hmmm - it's treating MD very differently to the LVM volume - different numbers of AGs, different sunit/swdith. Did you build xfsprogs yourself? Is it linked against libblkid or libdisk?
I should clarify - the first set was created with xfsprogs 3.0.0 and second was done with xfsprogs 3.1.3, so I wondered if the default ag count had changed.
Given a 12TB array, would I be better using 32?
I did build 3.1.3. Is libblkid preferable? I note it is defaulted off in configure and I used the default configuration.
l appear to have e2fsprogs and devel packages installed (Fedora 11), for libblkid, but when I enable and try to build, it fails:
[CC] xfs_mkfs.oxfs_mkfs.c: In function ‘check_overwrite’:
xfs_mkfs.c:298: error: ‘blkid_probe’ undeclared (first use in this function) xfs_mkfs.c:298: error: (Each undeclared identifier is reported only once xfs_mkfs.c:298: error: for each function it appears in.)
xfs_mkfs.c:298: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:321: error: ‘pr’ undeclared (first use in this function) xfs_mkfs.c:321: warning: implicit declaration of function ‘blkid_new_probe_from_filename’ xfs_mkfs.c:325: warning: implicit declaration of function ‘blkid_probe_enable_partitions’ xfs_mkfs.c:329: warning: implicit declaration of function ‘blkid_do_fullprobe’ xfs_mkfs.c:345: warning: implicit declaration of function ‘blkid_probe_lookup_value’ xfs_mkfs.c:362: warning: implicit declaration of function ‘blkid_free_probe’ xfs_mkfs.c: In function ‘blkid_get_topology’:
xfs_mkfs.c:372: error: ‘blkid_topology’ undeclared (first use in this function) xfs_mkfs.c:372: error: expected ‘;’ before ‘tp’
xfs_mkfs.c:373: error: ‘blkid_probe’ undeclared (first use in this function) xfs_mkfs.c:373: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:381: error: ‘pr’ undeclared (first use in this function) xfs_mkfs.c:385: error: ‘tp’ undeclared (first use in this function) xfs_mkfs.c:385: warning: implicit declaration of function ‘blkid_probe_get_topology’ xfs_mkfs.c:397: warning: implicit declaration of function ‘blkid_topology_get_minimum_io_size’ xfs_mkfs.c:400: warning: implicit declaration of function ‘blkid_topology_get_optimal_io_size’ xfs_mkfs.c:403: warning: implicit declaration of function ‘blkid_probe_get_sectorsize’ xfs_mkfs.c:406: warning: implicit declaration of function ‘blkid_topology_get_alignment_offset’ gmake[2]: *** [xfs_mkfs.o] Error 1
gmake[1]: *** [mkfs] Error 2
make: *** [default] Error 2
Quoted 3 lines
Or it might be that LVM is not exporting the characteristic of the underlying volume. Can you check if there are different parameter values exported by the two devices in /sys/block/<dev>/queue?
They look the same.
Quoted 8 lines
Quoted 3 lines
Limited testing using dd and bonnie++ shows no difference in write performance whether I use sunit=64/swidth=896 or sunit=0/swidth=0 on the lv.
These benchmarks won't realy show any difference on an empty filesystem. It will have an impact on how the filesystems age and how well aligned the IO will be to the underlying device under more complex workloads...
I figured I'd go with the geometry specified.
Regards,
Richard
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Richard Scobie
I did build 3.1.3. Is libblkid preferable? I note it is defaulted off in configure and I used the default configuration.
l appear to have e2fsprogs and devel packages installed (Fedora 11), for libblkid, but when I enable and try to build, it fails:
[CC] xfs_mkfs.o
xfs_mkfs.c: In function ‘check_overwrite’:
xfs_mkfs.c:298: error: ‘blkid_probe’ undeclared (first use in this function)
yup F11 doesn't have all the nice blkid topology bits; F11 still has libblkid in e2fsprogs, from before it was liberated to util-linux-ng.
(I wonder if we shouldn't just convert xfsprogs over to use the ioctls...)
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Eric Sandeen
yup F11 doesn't have all the nice blkid topology bits; F11 still has libblkid in e2fsprogs, from before it was liberated to util-linux-ng.
(I wonder if we shouldn't just convert xfsprogs over to use the ioctls...)
Thanks both for your help. For various reasons, I'll just rely on using the non-topology aware version.
In the future this lv will be grown in multiples of 256K chunk, 16 drive RAID6 arrays, so am I correct in thinking that the sunit/swidth parameter can stay the same as it is expanded?
I am thinking that XFS allocates ag's across all the space and that it will only be writing to any one array at a time, or would there be more configured if mkfs.xfs were aware that it was being created on say, 2 x 16 RAID6 arrays?
This is based on backing up the first array prior to expansion, adding the second and mkfs.xfs, as I imagine just expanding the fs over the second would result in non optimal performance.
Regards,
Richard
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Richard Scobie put forth on 9/10/2010 4:42 PM:
Quoted 3 lines
In the future this lv will be grown in multiples of 256K chunk, 16 drive RAID6 arrays, so am I correct in thinking that the sunit/swidth parameter can stay the same as it is expanded?
What is the reasoning behind adding so many terabytes under a single filesystem?
Do you _need_ all of it under a single mount point? If not, or even if you do, for many reasons, it may very well be better to put a single filesystem directly on each RAID6 array without using LVM in the middle and simply mount each filesystem at a different point, say:
/data
/data/array1
/data/array2
/data/array3
/data/array4
This method can minimize damage and downtime when an entire array is knocked offline. We just had a post yesterday where a SATA cable was kicked loose and took 5 drives down of a 15 drive md RAID6 set, killing the entire filesystem. If that OP had setup 3x5 drive arrays with 3 filesystems, the system could have continued to run in a degraded fashion, depending on his application data layout across the filesystems. If done properly, you lose an app or two, not all of them.
This method also eliminates xfs_growfs performance issues such as what you're describing because you're never changing the filesystem layout when adding new arrays to the system.
In summary, every layer of complexity added to the storage stack increases the probability of failure. As my grandmother was fond of saying, "Don't put all of your eggs in one basket." It was salient advice on the farm 80 years ago, and it's salient advice today with high technology.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs