Skip to contentSkip to conversation

Limits to growth

DISCUSSION Chris Dunlop 2 participants 3 messages Apr 14, 2022

lore
Share & tools
lore
#0 lore raw

Hi,

I have a nearly full 30T xfs filesystem that I need to grow significantly, e.g. to, say, 256T, and potentially further in future, e.g. up to, say, 1PB. Alternatively at some point I'll need to copy a LOT of data from the existing fs to a newly-provisioned much larger fs. If I'm going to need to copy data around I guess it's better to do it now, before there's a whole lot more data to copy.

According to Dave Chinner:

https://www.spinics.net/lists/linux-xfs/msg20084.html
Rule of thumb we've stated every time it's been asked in the past 10-15 years is "try not to grow by more than 10x the original size".

It's also explained the issue is the number of AGs.

Is it ONLY the number of AGs that's a concern when growing a fs?

E.g. for a fs starting in the 10s of TB that may need to grow substantially (e.g. >=10x), is it advisable to simply create it with the maximum available agsize, and you can then grow to whatever multiple without worrying about XFS getting ornery?

Of course as Dave explains further in the thread it would probably be better to just start with XFS on a large thin provisioned volume in the first place, but that's not where I am currently. Sigh.

Looking my fs and just considering the number of AGs (agcount)...

My original fs has:

meta-data=xxxx           isize=512    agcount=32, agsize=244184192 blks
         =               sectsz=4096  attr=2, projid32bit=1
         =               crc=1        finobt=1, sparse=1, rmapbt=1
         =               reflink=1    bigtime=0 inobtcount=0
data     =               bsize=4096   blocks=7813893120, imaxpct=5
         =               sunit=128    swidth=512 blks
naming   =version 2      bsize=4096   ascii-ci=0, ftype=1
log      =internal log   bsize=4096   blocks=521728, version=2
         =               sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none           extsz=4096   blocks=0, rtextents=0

If I do a test xfs_grow to 256T, it shows:

metadata=xxxxx isize=512 agcount=282, agsize=244184192 blks

Creating a new fs on 256T, I get:

metadata=xxxxx isize=512 agcount=257, agsize=268435328 blks

So growing the fs from 30T to 256T I end up with an agcount ~10% larger (and agsize ~10% smaller) than creating a 256T fs from scratch.

Just for the exercise, creating a new FS on 1P (i.e. 33x the current fs) gives:

metadata=xxxxx isize=512 agcount=1025, agsize=268435328 blks

I.e. it looks like for this case the max agsize is 268435328 blocks. So even if the current fs were to grow to a 1P or more, e.g. 30x - 60x original, I'm still only going to be ~10% worse off in terms of agcount than creating a large fs from scratch and copying all the data over.

Is that really going to make a significant difference?

Cheers,

Chris

#1 lore raw
Quoted 4 lines · Chris Dunlop

Hi,

I have a nearly full 30T xfs filesystem that I need to grow significantly, e.g. to, say, 256T, and potentially further in future, e.g. up to, say, 1PB.

That'll be fun. :)

Quoted 7 lines

Alternatively at some point I'll need to copy a LOT of data from the existing fs to a newly-provisioned much larger fs. If I'm going to need to copy data around I guess it's better to do it now, before there's a whole lot more data to copy.

According to Dave Chinner:

Quoted 7 lines

https://www.spinics.net/lists/linux-xfs/msg20084.html
Rule of thumb we've stated every time it's been asked in the past 10-15
years is "try not to grow by more than 10x the original size".

It's also explained the issue is the number of AGs.

Is it ONLY the number of AGs that's a concern when growing a fs?

No.

Quoted 4 lines

E.g. for a fs starting in the 10s of TB that may need to grow substantially (e.g. >=10x), is it advisable to simply create it with the maximum available agsize, and you can then grow to whatever multiple without worrying about XFS getting ornery?

If you start with anything greater 4-32TB, there's a good chance you've already got maximally sized AGs....

Quoted 5 lines

Looking my fs and just considering the number of AGs (agcount)...

My original fs has:

meta-data=xxxx isize=512 agcount=32, agsize=244184192 blks

Which is just short of maximally sized AGs. There's nothing to be gained by reformatting to larger AGs here.

Quoted 10 lines
         =               sectsz=4096  attr=2, projid32bit=1
         =               crc=1        finobt=1, sparse=1, rmapbt=1
         =               reflink=1    bigtime=0 inobtcount=0
data     =               bsize=4096   blocks=7813893120, imaxpct=5
         =               sunit=128    swidth=512 blks
naming   =version 2      bsize=4096   ascii-ci=0, ftype=1
log      =internal log   bsize=4096   blocks=521728, version=2
         =               sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none           extsz=4096   blocks=0, rtextents=0
Quoted 7 lines

If I do a test xfs_grow to 256T, it shows:

metadata=xxxxx isize=512 agcount=282, agsize=244184192 blks

Creating a new fs on 256T, I get:

metadata=xxxxx isize=512 agcount=257, agsize=268435328 blks

Yup.

Quoted 2 lines

So growing the fs from 30T to 256T I end up with an agcount ~10% larger (and agsize ~10% smaller) than creating a 256T fs from scratch.

Yup.

Quoted 4 lines

Just for the exercise, creating a new FS on 1P (i.e. 33x the current fs) gives:

metadata=xxxxx isize=512 agcount=1025, agsize=268435328 blks

Yup.

Quoted 1 line

I.e. it looks like for this case the max agsize is 268435328 blocks.

Yup.

Quoted 4 lines

So even
if the current fs were to grow to a 1P or more, e.g. 30x - 60x original, I'm still only going to be ~10% worse off in terms of agcount than creating a large fs from scratch and copying all the data over.

Yup.

Quoted 1 line

Is that really going to make a significant difference?

No.

But there will be significant differences. e.g. think of the data layout and free space distribution of a 1PB filesystem that it is 90% full and had it's data evenly distributed throughout it's capacity. Now consider the free space distribution of a 100TB filesystem that has been filled to 99% and then grown by 100TB nine times to a capacity of 90% @ 1PB. Where is all the free space?

That's right - the free space is only in the region that was appended in the last 100TB grow operation. IOWs, 90% of the AGs are completley full, and the newly added 10% are compeltely empty.

However, the allocation algorithms do linear target increments and linear scans over *all AGs* trying to distribute the allocation across the entire filesystem and to find the best available free space for allocations. When you have hundreds of AGs and only 10% of them have usable free space, this becomes a problem. e.g. if the locality algorithm targets low numbered AGs that are full (and it will because the target increments and wraps in a linear fashion), then it might be scanning hundreds of AGs before it finds one of the recently added high numbered AGs with a big enough free space to allocate from.

Then consider that it is not unreasonable for the filesystem to hit this case for thousands of consecutive allocations at a time (e.g. untarring a tarball full of small files such as a kernel source tree will trigger this), maybe even occur for every single allocation over a time span of minutes or even hours.

IOWs, the scanning algorithms don't really scale to large numbers of AGs when most of the AGs are full and cannot be allocated from, and repeatedly growing full filesystems pushes the algorithms into highly undesirable corner cases much, much faster than filesystems that started off with that capacity,,,

IOWs, growing by more than 10x really starts to push the limits of the algorithms regardless of the AG count it results in. It's not a capacity thing - it's a reflection of the number of AGs with usable free space in them and the algorithms used to find free space in those AGs.

The algorithms can be fixed, but it's not been an important issue to solve because so few people are using grow[*] in this manner - growing once or twice is generally as much as occurs over the life of a typical production filesysetm...

Cheers,

Dave.

[*] Now, if you have a 2GB filesystem and you grow it to several TB (that's a nasty antipattern we see quite frequently in cloud deployments) then having 10,000+ tiny AGs has these linear scan problems as well as all sorts of other scalability issues related to the sheer number of AGs, but that's a different set of large ag count problems....

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw

Hi Dave,

Quoted 7 lines · Dave Chinner
Quoted 4 lines · Chris Dunlop

Hi,

I have a nearly full 30T xfs filesystem that I need to grow significantly, e.g. to, say, 256T, and potentially further in future, e.g. up to, say, 1PB.

That'll be fun. :)

Yeah, good problem to have.

Quoted 15 lines
Quoted 1 line

So even if the current fs were to grow to a 1P or more, e.g. 30x - 60x original, I'm still only going to be ~10% worse off in terms of agcount than creating a large fs from scratch and copying all the data over.

Yup.

Quoted 1 line

Is that really going to make a significant difference?

No.

But there will be significant differences. e.g. think of the data layout and free space distribution of a 1PB filesystem that it is 90% full and had it's data evenly distributed throughout it's capacity. Now consider the free space distribution of a 100TB filesystem that has been filled to 99% and then grown by 100TB nine times to a capacity of 90% @ 1PB. Where is all the free space?

Quoted 3 lines

That's right - the free space is only in the region that was appended in the last 100TB grow operation. IOWs, 90% of the AGs are completley full, and the newly added 10% are compeltely empty.

Yep. But growing from 30T (@ 89% full), to 256T, then possibly to 512T and then 1P shouldn't suffer from this issue as much - perhaps assisted by growing when it reaches, say, 60-70% rather than waiting till 90%.

In this case I might be in a decent position: the data is generally large-ish backup files and the older ones get deleted as they age out, so that should free up a significant amount of space in the currently-near-full AGs over time. It sounds like this, in conjunction with the balancing of the allocation algorithms, should end up homogenising the usage over all the AGs.

Or, if there's a chance it'll get to 1P, would it be better to grow it out that far now (on thin storage - ceph rbd in this case)?

On the other hand, I may want newer XFS feature goodies before I need 1P in this thing.

Quoted 5 lines

IOWs, growing by more than 10x really starts to push the limits of the algorithms regardless of the AG count it results in. It's not a capacity thing - it's a reflection of the number of AGs with usable free space in them and the algorithms used to find free space in those AGs.

I'm not sure if 89% full has already put me behind, but outside of that, doing a grow at, say, 60-70% full rather than 90+%, in conjunction with a decent amount of data turnover, should reduce or remove this problem, no?

I.e. would you say, growing by more (possibly a lot more) than 10x would probably be ok *IF* you're starting with (near) maximally sized AGs, and growing the fs when it reaches, say, 60-70% full, and with data that has a reasonable turn over cycle?

Cheers,

Chris