What is a good stripe size?
Hi, idsfa!
[email protected] wrote 38 lines:
Quoted 9 lines · Urs Thuermann
Quoted 4 lines
Using the PVs on sda2 and sdb2 I created a single VG and I want to create striped LVs on it now. My questions is, how large should I choose the stripe size to achive optimal performance. If I choose it too large, I will probably loose the win of striping.
You lose the advantage of striping if the stripe size is on the order of the file size. You want stripes which will be narrower than most of the files you will be using.
I wonder if you are looking at a single file or general throughput here.
For a single file you may gain reading speed (writing is less critical as it is buffered); however with a stripe size below file size you will need to move the heads of both (or even more) disks, increasing latency[1], effectively slowing down reads unless you have fairly large files.
Often with more than one disk and randomly distributed files (e.g. through the use of stripes) random files are on different disks, so you can (best case) read as many files at the same time -- in parallel -- as you have disks. This is only true for sufficiently large stripes, though. Else a single file blocks many/all disks, forcing longer/extra seeks for the other files requested in parallel.
In conclusion (IMHO):
- small stripes increase the latency even for small reads, hurting throughput (and slowing the reads even when looking at a single file).
- sufficiently large stripes allow both parallel reads of small[2] files or accelerated reads (at the cost of extra -- in this case insignificant[3] -- seek time) of single large[2] files.
- Systems where the I/O is not the bottleneck, i.e. parallel reads or accelerated reads of large files won't help much, will not profit from stripes while still increasing[4] the risk of data loss through HD/controller failure.
- With lvm you could pmove the most accessed blocks so that they are spread over all disks, probably you could even split those disks in 2 parts: the fast 'begin' (outer edge) of the platter and most of the (slower) inner parts. This would have much the same effect of stripes, but would need more attention (you need to run a program, probably even inactivating the LV) and probably have finer granularity.
[1] Your seek time rises -- on the average -- the more heads are in use. With one head you get 1/3 full-seek time for a random head and file locations. With two heads you need to move both heads, chances are that one of them has a longer way than the other.
If you look at the first head, the seek time will be the same if the first head is further away --- which is more likely the further it is away, i.e. the higher the seek time already would be for a single disk system. If it is close (which happens and is part of the 1/3 _average_ seek time) then it is quite likely that the other head is further away -- thus the average seek time increases.
This is quite uncritical for large files, but with short files the seek time is greater than the read time. But only where the increased seek time is small against the read time a reduced read time can do reduce overall time -- that is, only large files will be faster.
[2] small: read time << seek time, fits in one stripe
large: read time >> seek time, does certainly not fit in
one stripe
[3] read time >> seek time
[4] all disks are 'single point of failure'. Most file systems do not like loosing spots all over the place. But then you do backup religiously, test your backups and have recovery plans in place, yes?
-Wolfgang
Quoted 10 lines
Quoted 9 lines · Urs Thuermann
Quoted 4 lines
Using the PVs on sda2 and sdb2 I created a single VG and I want to create striped LVs on it now. My questions is, how large should I choose the stripe size to achive optimal performance. If I choose it too large, I will probably loose the win of striping.
You lose the advantage of striping if the stripe size is on the order of the file size. You want stripes which will be narrower than most of the files you will be using.
Quoted 8 lines
I wonder if you are looking at a single file or general throughput here.
For a single file you may gain reading speed (writing is less critical as it is buffered); however with a stripe size below file size you will need to move the heads of both (or even more) disks, increasing latency[1], effectively slowing down reads unless you have fairly large files.
Assumes that the stripe blocks are not adjacent on their respective disk drives. The smaller stripe may have no penalty if the logical blocks are adjacent on the disk.
One of the main resons for using LVM at all is to avoid having to worry about any of this. If raw speed is a major consideration then use hardware RAID5 w/ strip size == I/O block size (e.g., 4 disks w/ 1K chunk and 4K filesystem block on linux). This avoids the "extra read" penalty and gives nice, distributed reads.
Quoted 5 lines
[1] Your seek time rises -- on the average -- the more heads are in use. With one head you get 1/3 full-seek time for a random head and file locations. With two heads you need to move both heads, chances are that one of them has a longer way than the other.
One advantage of striping is that the seek latency of one drive can be used for data I/O on another drive. If the LVM system does any sort of double-buffering then the striped system can negate/reduce the seek time.
This also leaves out the issue of journaled file systems, which may have data (or just meta-data) spread out all over the place -- leaveing you with fragmented reads even in the case of a small file.
Net result is that depending on CPU, bus, controller and disk hardware and their interactins with the file systems and particular type of I/O being performned the answer becomes "It Depends" :-)
In 15 years the only method I've found that works consistently is to try however many of the recommendations you can before comitting to any one of them. Benchmark them under realistic condidtions and one will usually be a bit better. At that point you can 'reverse engineer' why your particular conditions match that particular theory -- and probably learn a bit about how to improve your system as a result.
Quoted 4 lines
[4] all disks are 'single point of failure'. Most file systems do not like loosing spots all over the place. But then you do backup religiously, test your backups and have recovery plans in place, yes?
Ah, but it's so much more fun to fiture out how it all works at 3 in the morning with 20 users breathing fire down your back!
Quoted 5 lines · Wolfgang Weisselberg
For a single file you may gain reading speed (writing is less critical as it is buffered); however with a stripe size below file size you will need to move the heads of both (or even more) disks, increasing latency[1], effectively slowing down reads unless you have fairly large files.
True, but I was assuming 'typical' use, which means < 2G of small files (everything having to do with the OS) with the majority of the hard drive space being used for files in the megabyte+ range (multimedia, databases and so forth). I can agree with your logical argument that larger stripes are better for small files, but I would further argue that striping a partition of small files is the wrong way to go. If you are looking for a performance increase on reads for small files, you want to look@RAID1 (mirroring) or RAID 5 (if you have enough disk controllers) or a filesystem which optimises access to small files.
{ I neglected to mention in my example that I do not stripe my OS
partitions, but mirror them instead. I stripe my /home (where I do graphics and sound work) and /games. }
Quoted 4 lines
In conclusion (IMHO):
- small stripes increase the latency even for small reads, hurting throughput (and slowing the reads even when looking at a single file).
I'd have to change that to "increase the latency for small reads". For files >> stripe size, you will see no increase in latency. This was stated in your [1] endnote, as well.
Quoted 4 lines
- sufficiently large stripes allow both parallel reads of small[2] files or accelerated reads (at the cost of extra -- in this case insignificant[3] -- seek time) of single large[2] files.
Not argued. I was assuming large files. The answer to "how do I configure X" is almost always, "what do you want to do with it?"
Quoted 7 lines
- With lvm you could pmove the most accessed blocks so that they are spread over all disks, probably you could even split those disks in 2 parts: the fast 'begin' (outer edge) of the platter and most of the (slower) inner parts. This would have much the same effect of stripes, but would need more attention (you need to run a program, probably even inactivating the LV) and probably have finer granularity.
Something which would balance PEs within a VG based on their usage would be a lovely system tool to add to lvm. I can hardly wait for your program ;-)
Thanks for pointing out where I was making unspoken assumptions!
Signature · 1 line
$ fortune -m Kellen
Quoted 37 lines · Steven Lembark
Quoted 11 lines
Quoted 10 lines · Urs Thuermann
Quoted 4 lines
Using the PVs on sda2 and sdb2 I created a single VG and I want to create striped LVs on it now. My questions is, how large should I choose the stripe size to achive optimal performance. If I choose it too large, I will probably loose the win of striping.
You lose the advantage of striping if the stripe size is on the order of the file size. You want stripes which will be narrower than most of the files you will be using.
Quoted 8 lines
I wonder if you are looking at a single file or general throughput here.
For a single file you may gain reading speed (writing is less critical as it is buffered); however with a stripe size below file size you will need to move the heads of both (or even more) disks, increasing latency[1], effectively slowing down reads unless you have fairly large files.
Assumes that the stripe blocks are not adjacent on their respective disk drives. The smaller stripe may have no penalty if the logical blocks are adjacent on the disk.
One of the main resons for using LVM at all is to avoid having to worry about any of this. If raw speed is a major consideration then use hardware RAID5 w/ strip size == I/O block size (e.g., 4 disks w/ 1K chunk and 4K filesystem block on linux). This avoids the "extra read" penalty and gives nice, distributed reads.
Quoted 5 lines
[1] Your seek time rises -- on the average -- the more heads are in use. With one head you get 1/3 full-seek time for a random head and file locations. With two heads you need to move both heads, chances are that one of them has a longer way than the other.
Quoted 4 lines
One advantage of striping is that the seek latency of one drive can be used for data I/O on another drive. If the LVM system does any sort of double-buffering then the striped system can negate/reduce the seek time.
No double-buffering today :-(
LVM just redirects I/Os to the underlying devices (aka physical volumes).
Regards,
Heinz -- The LVM Guy --
Quoted 33 lines
This also leaves out the issue of journaled file systems, which may have data (or just meta-data) spread out all over the place -- leaveing you with fragmented reads even in the case of a small file.
Net result is that depending on CPU, bus, controller and disk hardware and their interactins with the file systems and particular type of I/O being performned the answer becomes "It Depends" :-)
In 15 years the only method I've found that works consistently is to try however many of the recommendations you can before comitting to any one of them. Benchmark them under realistic condidtions and one will usually be a bit better. At that point you can 'reverse engineer' why your particular conditions match that particular theory -- and probably learn a bit about how to improve your system as a result.
Quoted 4 lines
[4] all disks are 'single point of failure'. Most file systems do not like loosing spots all over the place. But then you do backup religiously, test your backups and have recovery plans in place, yes?
Ah, but it's so much more fun to fiture out how it all works at 3 in the morning with 20 users breathing fire down your back!
Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-