What is a good stripe size?
A message that is not in the archive (sent off-list or not yet received): 134718885.992837796416.JavaMail.root@boots
Hi, idsfa!
[email protected] wrote 63 lines:
Quoted 11 lines · Wolfgang Weisselberg
Quoted 5 lines
For a single file you may gain reading speed (writing is less critical as it is buffered); however with a stripe size below file size you will need to move the heads of both (or even more) disks, increasing latency[1], effectively slowing down reads unless you have fairly large files.
True, but I was assuming 'typical' use, which means < 2G of small files (everything having to do with the OS) with the majority of the hard drive space being used for files in the megabyte+ range (multimedia, databases and so forth).
Databases can be special cases here, they often only read small parts of their storage files (retriving a 3-char fields e.g.). But then many Databases want raw access and probably will do striping themselves, too.
A maildir or MH-style central mail spool (e.g. qmail) will contain lots and lots of small to medium files -- one file per email.
Further, reiserfs is pretty good with many many files, this will (over some development time) lead to smaller overall file size as applications will be more often programmed not to aggregate data into bigger files. But this is something which will not trouble us much now.
Quoted 3 lines
I can agree with your logical argument that
larger stripes are better for small files, but I would further argue that striping a partition of small files is the wrong way to go.
Again, this depends. If you need parallel access to small files, use stripes or even RAID10/15 (mirrored stripes/mirrored RAID5). This would be the case of a maildir imap server, for example. If this is no bottleneck -- let it be.
Quoted 7 lines
Quoted 4 lines
In conclusion (IMHO):
- small stripes increase the latency even for small reads, hurting throughput (and slowing the reads even when looking at a single file).I'd have to change that to "increase the latency for small reads". For files >> stripe size, you will see no increase in latency.
You will with small stripe sizes[5], but it's usually neglectable, since you finish earlier than with one disk.
Quoted 3 lines
Something which would balance PEs within a VG based on their usage would be a lovely system tool to add to lvm. I can hardly wait for your program ;-)
First, we need a fool^Wcrashproof, completely interruptible pvmove for active, being currently read from and written to LVs. Once this is there, we need to have a pvmove which can be told the physical place to move to, else we can only spread the most accessed PEs over the PVs.
And at the moment pvmove can only partially move non-striped LVs.
Then the rest is simply ripping a balancing algorithm from somewhere and slap it into a wrapper. Data aquisition is already done via lvmsadc/lvmsar.
-Wolfgang
[5] Assume:
- 1st head closer to data than second head (about 50% of the time)
- small strip size (e.g. 4k)
- no bad fragmentation (normal case)
- idle I/O
You request the first 15k:
- head 1 seeks to stripe, so does head 2.
- head 1 arrives; the platter turns until the begin of the data; head 1 starts deliver data -- the whole strip (4k) and internal caching of the following strips begins.
- head 2 should now deliver the second 4k,
*but is still seeking.*
This is the added latency.
- head 2 finishes seeking, the platter turns until the begin of the data; head 2 delivers the second 4k.
- from here on the data is put out at 'double rate' from both disks.
With larger strip sizes the added latency occurs less often, as up to the whole strip size is read and can be delivered giving the other head more time to finish seeking.
With non-idle I/O the latency for gets worse, of course, even when reads are reordered to minimize the impact.
On Mon, Jun 18, 2001 at 12:33:18PM +0200, Wolfgang Weisselberg wrote:
...
Quoted 3 lines
First, we need a fool^Wcrashproof, completely interruptible pvmove for active, being currently read from and written to LVs.
I hope this is already there. Do your experiences suggest it isn't working for active pv's ?
Quoted 3 lines
Once this is there, we need to have a pvmove which
can be told the physical place to move to, else we can only spread the most accessed PEs over the PVs.
This is exactly what we're planning for the next version of LVM. The moving of extents will be performed by the kernel rather than in user space, the ioctl interface will allow the user to specify a list of pe movements.
Quoted 3 lines
Then the rest is simply ripping a balancing algorithm from somewhere and slap it into a wrapper. Data aquisition is already done via lvmsadc/lvmsar.
Yes, a little Perl script to process the usage stats and then create a new map.
- Joe
Joe Thornber ([email protected]) wrote 33 lines:
Quoted 8 lines · Wolfgang Weisselberg
Quoted 4 lines
First, we need a fool^Wcrashproof, completely interruptible pvmove for active, being currently read from and written to LVs.
I hope this is already there. Do your experiences suggest it isn't working for active pv's ?
man pvmove:
[...]
You can move physical extents in use but make sure you have an current backup in case of a system crash while moving!!!
Now, this does *not* look like *crashproof*, does it? This is from the CVS, btw, dated 2001-06-10.
And I am unwilling to have a tool run automatically (at night?) that - upon a crash - can destroy whole partitions. I could live with a 'copy, repeat if original was changed, lock LE, update maps on HD, unlock LE (on new PE)' thing.
I have used pmove about once in ernest, and my, moving 35 Gigs over to a new HD takes time! And that does not look like a HD bottleneck...
Quoted 4 lines
This is exactly what we're planning for the next version of LVM. The moving of extents will be performed by the kernel rather than in user space, the ioctl interface will allow the user to specify a list of pe movements.
Sounds interesting.
Quoted 6 lines
Quoted 3 lines
Then the rest is simply ripping a balancing algorithm from somewhere and slap it into a wrapper. Data aquisition is already done via lvmsadc/lvmsar.
Yes, a little Perl script to process the usage stats and then create a new map.
There it might be interesting to know which blocks are accessed in sequence. Then again it might not, I am not an expert there and haven't done the maths for that.
-Wolfgang
Quoted 8 lines · Wolfgang Weisselberg
man pvmove:
[...]
You can move physical extents in use but make sure you have an current backup in case of a system crash while moving!!!Now, this does *not* look like *crashproof*, does it? This is from the CVS, btw, dated 2001-06-10.
Heinz,
Do we update the metadata after each individual extent is moved ? Or at the end of the whole pvmove ?
I'd assumed we were doing the former, in which case this comment from the man page is unduly pesimistic.
- Joe
Quoted 7 lines · Joe Thornber
Quoted 6 lines · Wolfgang Weisselberg
man pvmove:
[...]
You can move physical extents in use but make sure you have an current backup in case of a system crash while moving!!!
Quoted 7 lines
Quoted 2 lines
Now, this does *not* look like *crashproof*, does it? This is from the CVS, btw, dated 2001-06-10.
Heinz,
Do we update the metadata after each individual extent is moved ? Or at the end of the whole pvmove ?
1st case.
Metadata is updated after every single moved extent.
The only reason why the man page says the above is the non atomic nature of the metadata updates (2 block devices need to be updated; source and destination) in case of the version 1 metadata layout.
With v2 (transaction oriented updates) we should be fine and can remove the above warning.
Quoted 3 lines
I'd assumed we were doing the former, in which case this comment from the man page is unduly pesimistic.
Yes, it is somehow pesimistic but disks can break, power can be interupted or a million other things can happen which let the source device metadata update make it but *not* the destination device update.
We could eventually recover from that situation by updating the destination before the source and checking double entries for the same logical extent on the source and destination device at VG activation time in order to remove one of the two entries.
Heinz
Quoted 7 lines
- Joe
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-