RFE? Really power of 2? extents, chunks and raid alignment
I was trying to figure out why I didn't get performance out of my RAID, except when operating on really large files where it's many stripes v. 1 stripe.
I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested stripesize of 64k, so a stripe-width of 768k.
Some issues that have been nagging me though are getting my allocations lined up on 768k boundaries. xfs is no prob -- tell it 64k and 12 and it does it.
But I just realized that lvm doesn't really tell me where it is aligning things and worse, only lets me align both chunksizes for lv's and extents for vg's
in powers of 2. Um... Not a multiple of 4k? or 64k?
Am I wrong in thinking this would tend to give me both pv's and lv's that are very likely NOT to be stripe-width aligned, but, worse, not stripewidth alignable, at all.
This would mean that a high performance file system aware of RAID stripe with that tries to allocate chunks starting on a 768k boundary are likely to just get it completely wrong? No?
Or what am I missing?
Thanks...
-linda
Quoted 10 lines · Linda Walsh
I was trying to figure out why I didn't get performance out of my RAID, except when operating on really large files where it's many stripes v. 1 stripe.
I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested stripesize of 64k, so a stripe-width of 768k.
Quoted 6 lines
Some issues that have been nagging me though are getting my allocations lined up on 768k boundaries. xfs is no prob -- tell it 64k and 12 and it does it.
But I just realized that lvm doesn't really tell me where it is aligning things
pvs --units s -o pv_name,vg_name,pe_start
pvcreate: --metadatasize, --dataalignment, --dataalignmentoffset
vgcreate: --physicalextentsize
Quoted 7 lines
and worse, only lets me align both chunksizes for lv's and extents for vg's
in powers of 2. Um... Not a multiple of 4k? or 64k?Am I wrong in thinking this would tend to give me both pv's and lv's that are very likely NOT to be stripe-width aligned, but, worse, not stripewidth alignable, at all.
Quoted 7 lines
This would mean that a high performance file system aware of RAID stripe with
that tries to allocate chunks starting on a 768k boundary are likely to just
get it completely wrong? No?Or what am I missing?
If you get the pvcreate --dataalignment right,
Any PE size 256k and larger will be fine,
just make sure you allocate your LV sizes in multiple of 3 PEs.
lvcreate -n lv_bla -l $(( (lv_size_kb / pe_size_kb + 2) / 3 * 3 )) vg_foo
Signature · 3 lines
: Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com
Quoted 5 lines · Lars Ellenberg
pvs --units s -o pv_name,vg_name,pe_start
pvcreate: --metadatasize, --dataalignment, --dataalignmentoffset
vgcreate: --physicalextentsize
---
BTW, thanks for this ... I missed it earlier -- and have been sure
to save away a
copy for future reference...
now if only disk prices would come back down (or rather (*sardonic
laugh*), the dollar
value would go back up)......
Quoted 23 lines · Linda Walsh
I was trying to figure out why I didn't get performance out of my RAID, except when operating on really large files where it's many stripes v. 1 stripe.
I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested stripesize of 64k, so a stripe-width of 768k.
Some issues that have been nagging me though are getting my allocations lined up on 768k boundaries. xfs is no prob -- tell it 64k and 12 and it does it.
But I just realized that lvm doesn't really tell me where it is aligning things
and worse, only lets me align both chunksizes for lv's and extents for vg's
in powers of 2. Um... Not a multiple of 4k? or 64k?Am I wrong in thinking this would tend to give me both pv's and lv's that are very likely NOT to be stripe-width aligned, but, worse, not stripewidth alignable, at all.
Quoted 7 lines
This would mean that a high performance file system aware of RAID stripe with
that tries to allocate chunks starting on a 768k boundary are likely to just
get it completely wrong? No?Or what am I missing?
The DM update for the 3.6 merge window adds non power of 2 support in the kernel (for the stripe and thin-pool targets).
So the lvm2 constraints that require a power of 2 chunksize will be relaxed very shortly.
Quoted 4 lines · Mike Snitzer
Quoted 4 lines
I have 12 data disks in a RAID 50 (3 RAID5's in a RAID0) and use a suggested stripesize of 64k, so a stripe-width of 768k.
I must state this correction -- to the above -- I was thinking I had to use the least common multiplier -- but in reality, I _think_ (subject to later revision, both -- reality and what I think, ;-)), I should be able to use the greatest common denominator and that should work equally well.
Nevertheless, this:
Quoted 7 lines
The DM update for the 3.6 merge window adds non power of 2 support in the kernel (for the stripe and thin-pool targets).
So the lvm2 constraints that require a power of 2 chunksize will be relaxed very shortly.
is great news, as I'm sure that being able to synchronize stripes to chunksize AND extent size, would allow better performance for RAID aware file systems...
I.e. As it is now, I need to have my extents line up, my chunks line up AND my file system block allocations line up....*ouch*....
If any of those start at 'fixed offsets' from the beginnning of their space and that fixed size isn't a multiple of the extent size (for VG's) OR chunksize for LV's, then all of the VG's/LV's will won't line up -- this is only an issue if you have multiple VG's per PV (i.e. the PV's will use the same physical disks, and the LV's would also use the same disks, but what you thought was an aligned database mirrored in 1 partition might be completely non-aligned if anything under it is
misaligned.
I very well do NOT know, the full impact of such -- only that it would cause unpredictable or variable performance, which as an engineer, bothers me.