xfsprogs/mkfs: prototype XFS image mode format for scalable AG growth
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/
mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
mkfs/xfs_mkfs.c +5 -6
@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,
@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,
@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);
@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();
Signature · 1 line
2.45.0
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and xfs_repair warnings are concerning, but to that, I have these things to say:
a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
b. We can turn off the xfs_repair hostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
I think this is worth prototyping a bit more. Do you?
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 5 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.
Quoted 8 lines
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional GROWFS_AGSIZE ioctl without much additional fuss.
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update xfs_growfs to look at the current block device size, make a decision on an ideal AG size from that, and run an ioctl(GROWFS_AGSIZE, agsize) followed by the typical ioctl(GROWFS_DATA, bdev_size). The GROWFS_AGSIZE can either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let GROWFS_AGSIZE fail if any such unsupported fields are set. Worst case we just fall back to existing behavior.
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..
Quoted 57 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 106 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 5 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.Quoted 8 lines
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Quoted 3 lines
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.
What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e. xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.
Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.
I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..
<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 107 lines · Darrick J. Wong
Quoted 106 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 13 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Quoted 9 lines
Quoted 3 lines
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Interesting, that seems notably larger than I'd expect. Wouldn't this concern you if you just ended up with a 37GB single AG fs?
Do you know how much of that might be free space on the image side? I.e., I'm wondering if the size is that large because you really need the space for data, or rather as a workaround where it could otherwise be formatted much smaller if the deployment side could grow it out to well beyond 37GB without having issues.
With a 37GB 4xAG fs, that's still closer to ~9GB AGs, which doesn't seem all that bad from a grow perspective. IIRC the insane filesystems I've seen tend to have much smaller (< 1GB maybe?) AG sizes. Have you seen grow issues with these ~9GB AG size fs'?
Quoted 31 lines
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.
Quoted 4 lines
What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e.
xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.
Either way seems reasonable. I think we're mostly talking about the same thing. For a functional prototype I'd be more concerned with just showing whether it works and then if so, whether it's useful/flexible enough for the intended use cases.
Brian
Quoted 102 lines
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 112 lines · Brian Foster
Quoted 111 lines · Darrick J. Wong
Quoted 109 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 5 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.Quoted 8 lines
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
Quoted 8 lines
Quoted 5 lines
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Interesting, that seems notably larger than I'd expect. Wouldn't this concern you if you just ended up with a 37GB single AG fs?
Not that much if I had fsverity to ensure that the deployed files hadn't been tampered with.
Quoted 1 line
Do you know how much of that might be free space on the image side?
$ xfs_info /
meta-data=/dev/mapper/ocivolume-root isize=512 agcount=4, agsize=2324736 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=9298944, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=4540, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
$ df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ocivolume-root 37177616 14264712 22912904 39% /So I think the 9G AGs aren't that much of a problem. OTOH
Quoted 4 lines
I.e., I'm wondering if the size is that large because you really need the space for data, or rather as a workaround where it could otherwise be formatted much smaller if the deployment side could grow it out to well beyond 37GB without having issues.
It depends on the workload -- vanilla OL images only give you a rootfs and assume that you'll create a separate higher iops block volume for actual data storage. $database automates provisioning and autotuning the separate block device for you.
Another problem is that we don't necessarily know how many CPUs the system will have. I've seen a system with ~200 cores all pounding on a fs with four AGs (oops! lots of AGF contention!) or 10000 AGs (all cpus pounding on the radix tree).
IOWs, the deploy image could be a lot smaller, but apparently they're 48GB in size. My guess is that these are sparse images, since the billable size is about a tenth of that.
Quoted 4 lines
With a 37GB 4xAG fs, that's still closer to ~9GB AGs, which doesn't seem all that bad from a grow perspective. IIRC the insane filesystems I've seen tend to have much smaller (< 1GB maybe?) AG sizes. Have you seen grow issues with these ~9GB AG size fs'?
Not on our homegrown images. The problems crop up when people try to import images from outside $cloud and generate those minified images with 1G AGs. Unfortunately some of those people aren't even outside the org, but I'll get to them... ;)
Quoted 36 lines
Quoted 31 lines
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.Quoted 4 lines
What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e.
xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.
Quoted 4 lines
Either way seems reasonable. I think we're mostly talking about the same thing. For a functional prototype I'd be more concerned with just showing whether it works and then if so, whether it's useful/flexible enough for the intended use cases.
Agreed!
Quoted 106 lines
Brian
Quoted 102 lines
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 119 lines · Darrick J. Wong
Quoted 118 lines · Brian Foster
Quoted 116 lines · Darrick J. Wong
Quoted 106 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 5 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.Quoted 8 lines
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Quoted 3 lines
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Quoted 6 lines
Quoted 2 lines
Interesting, that seems notably larger than I'd expect. Wouldn't this concern you if you just ended up with a 37GB single AG fs?
Not that much if I had fsverity to ensure that the deployed files hadn't been tampered with.
Well that covers the deployment part, but personally for anything more than a scratch/test vm I'd be a little worried about repairability and not having enough redundancy and whatnot for ongoing usage.
Quoted 17 lines
Quoted 1 line
Do you know how much of that might be free space on the image side?
$ xfs_info / meta-data=/dev/mapper/ocivolume-root isize=512 agcount=4, agsize=2324736 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=9298944, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=4540, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 $ df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/ocivolume-root 37177616 14264712 22912904 39% /
Quoted 2 lines
So I think the 9G AGs aren't that much of a problem. OTOH
Yeah.. so even if this weren't formatted as a larger sparse image, say the image file size were reduced to 15G or so, even something like a 2xAG w/ 10GB agsize seems plausible as an initial format. That doesn't require any growing tricks at all, and even if it never grows and performance is not a huge concern, you at least have some basic superblock redundancy.
That raises yet another thought.. I wonder if we could incrementally improve things by say just modernizing some of the format time heuristics for image files. I.e., if we see we're formatting a regular file in the tens of GBs in size, perhaps having a 2xAG minimum with a larger agsize is more appropriate than the typical 4xAG geometry we'd otherwise use on a 20GB hard drive from 25+ years ago.
Even if that custom heuristic were buried behind a 'mkfs.xfs --cloud ...' param, that would at the very least give us something to evangelize to users as these "why does my 3TB 128MB agsize fs suck?" reports continue to come in..
Quoted 10 lines
Quoted 4 lines
I.e., I'm wondering if the size is that large because you really need the space for data, or rather as a workaround where it could otherwise be formatted much smaller if the deployment side could grow it out to well beyond 37GB without having issues.
It depends on the workload -- vanilla OL images only give you a rootfs and assume that you'll create a separate higher iops block volume for actual data storage. $database automates provisioning and autotuning the separate block device for you.
Right.. so I assume these filesystems can be sized appropriately from the start.
Quoted 5 lines
Another problem is that we don't necessarily know how many CPUs the system will have. I've seen a system with ~200 cores all pounding on a fs with four AGs (oops! lots of AGF contention!) or 10000 AGs (all cpus pounding on the radix tree).
Indeed.
Quoted 9 lines
IOWs, the deploy image could be a lot smaller, but apparently they're 48GB in size. My guess is that these are sparse images, since the billable size is about a tenth of that.
Quoted 4 lines
With a 37GB 4xAG fs, that's still closer to ~9GB AGs, which doesn't seem all that bad from a grow perspective. IIRC the insane filesystems I've seen tend to have much smaller (< 1GB maybe?) AG sizes. Have you seen grow issues with these ~9GB AG size fs'?
Quoted 5 lines
Not on our homegrown images. The problems crop up when people try to import images from outside $cloud and generate those minified images with 1G AGs. Unfortunately some of those people aren't even outside the org, but I'll get to them... ;)
Ok. This is the part that is still somewhat mysterious to me.
Given what we know about your internal images and how they are sized and whatnot, and expanding on the whole mkfs.xfs --cloud thought a bit, I'm wondering if we could just use something like that to enforce a minimum image size as well.
IOW, suppose 'mkfs.xfs --cloud ...' imposed a 20GB minimum file size and 10GB minimum AG size, documented that this is required for sane growability, and explained that the additional metadata over a 3GB image or whatever is mostly noise given sparse file image formats... do we think that would be sufficient to mostly avoid the problem? Would users adopt it if beat over the head enough with it when reporting problems with these ridiculous agcount filesystems? Hm?
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e.
xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.
Quoted 7 lines
Quoted 4 lines
Either way seems reasonable. I think we're mostly talking about the same thing. For a functional prototype I'd be more concerned with just showing whether it works and then if so, whether it's useful/flexible enough for the intended use cases.
Agreed!
Ok. It's on my todo list to clean up what I've hacked up wrt the whole grow agsize idea enough so it can be at least put on the list for discussion/posterity.. thanks.
Brian
Quoted 107 lines
Quoted 106 lines
Brian
Quoted 102 lines
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 123 lines · Brian Foster
Quoted 122 lines · Darrick J. Wong
Quoted 120 lines · Brian Foster
Quoted 107 lines · Darrick J. Wong
Quoted 106 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 13 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Quoted 9 lines
Quoted 3 lines
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Interesting, that seems notably larger than I'd expect. Wouldn't this concern you if you just ended up with a 37GB single AG fs?
Quoted 7 lines
Quoted 3 lines
Not that much if I had fsverity to ensure that the deployed files hadn't been tampered with.
Well that covers the deployment part, but personally for anything more than a scratch/test vm I'd be a little worried about repairability and not having enough redundancy and whatnot for ongoing usage.
<nod> Going all in on single-AG filesystems would likely create problems with old(er) versions of xfs_repair not liking the lack of redundancy. I concede that we shouldn't be doing that generally...
Quoted 20 lines
Quoted 17 lines
Quoted 1 line
Do you know how much of that might be free space on the image side?
$ xfs_info / meta-data=/dev/mapper/ocivolume-root isize=512 agcount=4, agsize=2324736 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=9298944, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=4540, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 $ df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/ocivolume-root 37177616 14264712 22912904 39% /Quoted 2 lines
So I think the 9G AGs aren't that much of a problem. OTOH
Quoted 6 lines
Yeah.. so even if this weren't formatted as a larger sparse image, say the image file size were reduced to 15G or so, even something like a 2xAG w/ 10GB agsize seems plausible as an initial format. That doesn't require any growing tricks at all, and even if it never grows and performance is not a huge concern, you at least have some basic superblock redundancy.
...to continue my line of thought above, what do you think about runt AGs? If say the gold master image has a 36G / partition, what if instead of formatting 2x18G AGs, we instead format a 35G AG0 and a 1G AG0? That way we always have two superblocks, and in the common case the runt won't stay a runt for long.
Quoted 6 lines
That raises yet another thought.. I wonder if we could incrementally improve things by say just modernizing some of the format time heuristics for image files. I.e., if we see we're formatting a regular file in the tens of GBs in size, perhaps having a 2xAG minimum with a larger agsize is more appropriate than the typical 4xAG geometry we'd otherwise use on a 20GB hard drive from 25+ years ago.
Yeah. Especially since these days the cloud rootfs storage is rather ssdlike.
Quoted 4 lines
Even if that custom heuristic were buried behind a 'mkfs.xfs --cloud ...' param, that would at the very least give us something to evangelize to users as these "why does my 3TB 128MB agsize fs suck?" reports continue to come in..
Yes!
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I.e., I'm wondering if the size is that large because you really need the space for data, or rather as a workaround where it could otherwise be formatted much smaller if the deployment side could grow it out to well beyond 37GB without having issues.
Quoted 8 lines
Quoted 5 lines
It depends on the workload -- vanilla OL images only give you a rootfs and assume that you'll create a separate higher iops block volume for actual data storage. $database automates provisioning and autotuning the separate block device for you.
Right.. so I assume these filesystems can be sized appropriately from the start.
<nod>
Quoted 17 lines
Quoted 5 lines
Another problem is that we don't necessarily know how many CPUs the system will have. I've seen a system with ~200 cores all pounding on a fs with four AGs (oops! lots of AGF contention!) or 10000 AGs (all cpus pounding on the radix tree).
Indeed.
Quoted 9 lines
IOWs, the deploy image could be a lot smaller, but apparently they're 48GB in size. My guess is that these are sparse images, since the billable size is about a tenth of that.
Quoted 4 lines
With a 37GB 4xAG fs, that's still closer to ~9GB AGs, which doesn't seem all that bad from a grow perspective. IIRC the insane filesystems I've seen tend to have much smaller (< 1GB maybe?) AG sizes. Have you seen grow issues with these ~9GB AG size fs'?
Quoted 7 lines
Quoted 5 lines
Not on our homegrown images. The problems crop up when people try to import images from outside $cloud and generate those minified images with 1G AGs. Unfortunately some of those people aren't even outside the org, but I'll get to them... ;)
Ok. This is the part that is still somewhat mysterious to me.
Yeah, the outside image thing is mysterious to us as well. Customers make those images, so we need sane(ish) defaults, or some serious evangelization of the --cloud option.
Quoted 5 lines
Given what we know about your internal images and how they are sized and whatnot, and expanding on the whole mkfs.xfs --cloud thought a bit, I'm wondering if we could just use something like that to enforce a minimum image size as well.
Quoted 7 lines
IOW, suppose 'mkfs.xfs --cloud ...' imposed a 20GB minimum file size and 10GB minimum AG size, documented that this is required for sane growability, and explained that the additional metadata over a 3GB image or whatever is mostly noise given sparse file image formats... do we think that would be sufficient to mostly avoid the problem? Would users adopt it if beat over the head enough with it when reporting problems with these ridiculous agcount filesystems? Hm?
I'd argue for a min sparse cloud image file size of 32G, and maybe even the one-and-runt AG setup above. But otherwise this sounds ok to me.
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 31 lines
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.Quoted 4 lines
What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e.
xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.Either way seems reasonable. I think we're mostly talking about the same thing. For a functional prototype I'd be more concerned with just showing whether it works and then if so, whether it's useful/flexible enough for the intended use cases.
Quoted 6 lines
Quoted 2 lines
Agreed!
Ok. It's on my todo list to clean up what I've hacked up wrt the whole grow agsize idea enough so it can be at least put on the list for discussion/posterity.. thanks.
NP.
--D
Quoted 111 lines
Brian
Quoted 107 lines
Quoted 106 lines
Brian
Quoted 102 lines
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0
Quoted 128 lines · Darrick J. Wong
Quoted 127 lines · Brian Foster
Quoted 125 lines · Darrick J. Wong
Quoted 112 lines · Brian Foster
Quoted 111 lines · Darrick J. Wong
Quoted 109 lines · Brian Foster
Quoted 62 lines · Darrick J. Wong
Quoted 61 lines · Brian Foster
Tweak a few checks to facilitate experimentation with an agcount=1 filesystem format with a larger agsize than the filesystem data size. The purpose of this is to POC a filesystem image mode format for XFS that better supports the typical cloud filesystem image deployment use case where a very small fs image is created and then immediately grown orders of magnitude in size once deployed to container environments. The large grow size delta produces filesystems with excessive AG counts, which leads to various other functional problems that eventually derive from this sort of pathological geometry.
To experiment with this patch, format a small fs with something like the following:
mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile>
Increase the underlying image file size, mount and grow. The filesystem will grow according to the format time AG size as if the AG was a typical runt AG on a traditional multi-AG fs.
This means that the filesystem remains with an AG count of 1 until fs size grows beyond AG size. Since the typical deployment workflow is an immediate very small -> very large, one-time grow, the image fs can set a reasonable enough default or configurable AG size (based on user input) that ensures deployed filesystems end up in a generally supportable geometry (i.e. with multiple AGs for superblock redundancy) before seeing production workloads.
Further optional changes are possible on the kernel side to help provide some simple guardrails against misuse of this mechanism. For example, the kernel could do anything from warn/fail or restrict runtime functionality for an insufficient grow. The image mode itself could set a backwards incompat feature bit that requires a mount option to enable full functionality (with the exception of growfs). More discussion is required to determine whether this provides a usable solution for the common cloud workflows that exhibit this problem and what the right interface and/or limitations are to ensure it is used correctly.
Not-Signed-off-by Brian Foster <[email protected]>---
Hi all,
This is a followup to the idea Darrick brought up in the expansion discussion here [1]. I poked through the code a bit and found it somewhat amusing how little was in the way of experimenting with this, so threw this against an fstests run over the weekend. I see maybe around ~10 or so test failures, most of which look like simple failures related to either not expecting agcount == 1 fs' or my generally hacky/experimental changes. There are a couple or so that require a bit more investigation to properly characterize before I would consider this fully sane.
I'm posting this separately from the expansion discussion to hopefully avoid further conflating the two. My current sense is that if this turns out to be a fundamentally workable approach, mkfs would more look something like 'mkfs --image-size 40g ...' and the kernel side may grow some optional guardrail logic mentioned above and in the previous discussion here [2], but others might have different ideas.
Quoted 9 lines
Quoted 3 lines
Darrick, you originally raised this idea and then Eric brought up some legitimate technical concerns in the expansion design thread. I'm curious if either of you have any further thoughts/ideas on this.
Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here.
Yeah.. I view this as kind of an incremental step in the broader expandfs topic to opportunistically solve a real (and lingering) problem with a minimal amount of complexity.
Quoted 10 lines
But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and
xfs_repairwarnings are concerning, but to that, I have these things to say:a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails.
This crossed my mind at one point as well. I thought these sort of image repo setups at least had per-image checksum files or whatever that would hopefully somewhat mitigate this sort of problem.
Quoted 5 lines
b. We can turn off the
xfs_repairhostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag.Quoted 8 lines
c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs.
Do you mean that the image size is 37GB, or it's grown to that size at some point during the deployment? If the latter, any idea on the ballpark size of a typical install image before it's deployed and grown?
Quoted 8 lines
Quoted 5 lines
I think the actual gold master images have ~37G root filesystems. I didn't choose to expand my stingy OL8 image at firstboot, and it has 4 AGs like you'd expect. Most of the time the deployment control panel tries to convince you to upgrade to 67G, if not ~200G.
Interesting, that seems notably larger than I'd expect. Wouldn't this concern you if you just ended up with a 37GB single AG fs?
Not that much if I had fsverity to ensure that the deployed files hadn't been tampered with.
Quoted 8 lines
Quoted 3 lines
Well that covers the deployment part, but personally for anything more than a scratch/test vm I'd be a little worried about repairability and not having enough redundancy and whatnot for ongoing usage.
<nod> Going all in on single-AG filesystems would likely create problems with old(er) versions of
xfs_repairnot liking the lack of redundancy. I concede that we shouldn't be doing that generally...
That might be reason enough for a feature bit. I've been waffling on the idea personally just because I'm a little concerned the existence of it might lead to overengineering of what ends up tied to it, but really if it existed for the mere purpose of backwards compatibility and letting newer ones spit out a warning for "absent or insufficient grow" situations, that seems pretty reasonable to me.
That said, I do think it's only really useful for the approach where we'd rely on growfs to fix up the fs before it should be used in production. If we ended up just changing mkfs to enforce larger AGs or whatever then it seems unnecessary IMO.
Quoted 27 lines
Quoted 20 lines
Quoted 17 lines
Quoted 1 line
Do you know how much of that might be free space on the image side?
$ xfs_info / meta-data=/dev/mapper/ocivolume-root isize=512 agcount=4, agsize=2324736 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=9298944, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=4540, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 $ df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/ocivolume-root 37177616 14264712 22912904 39% /Quoted 2 lines
So I think the 9G AGs aren't that much of a problem. OTOH
Quoted 6 lines
Yeah.. so even if this weren't formatted as a larger sparse image, say the image file size were reduced to 15G or so, even something like a 2xAG w/ 10GB agsize seems plausible as an initial format. That doesn't require any growing tricks at all, and even if it never grows and performance is not a huge concern, you at least have some basic superblock redundancy.
Quoted 6 lines
...to continue my line of thought above, what do you think about runt AGs? If say the gold master image has a 36G / partition, what if instead of formatting 2x18G AGs, we instead format a 35G AG0 and a 1G AG0? That way we always have two superblocks, and in the common case the runt won't stay a runt for long.
Definitely seems like a reasonable mkfs time approach to me.
I'm going to post a prototype of the dynamic AG size grow thing shortly. The heuristic I hacked in for that is basically to try for 4xAGs or otherwise set a minimum AG size of 4GB. IOW, a tiny fs that sees small, incremental grows would come out of single AG mode as soon as the fs size crosses 4GB plus the minimum number of blocks for a runt second AG.
Note that I'm not convinced any user would actually do such a thing with cloud fs images, so this is more just trying to be robust and cover the unexpected use cases than anything else. The larger point though is I don't see why mkfs couldn't do something similar with a runt AG if we wanted to go that route.
Quoted 56 lines
Quoted 6 lines
That raises yet another thought.. I wonder if we could incrementally improve things by say just modernizing some of the format time heuristics for image files. I.e., if we see we're formatting a regular file in the tens of GBs in size, perhaps having a 2xAG minimum with a larger agsize is more appropriate than the typical 4xAG geometry we'd otherwise use on a 20GB hard drive from 25+ years ago.
Yeah. Especially since these days the cloud rootfs storage is rather ssdlike.
Quoted 4 lines
Even if that custom heuristic were buried behind a 'mkfs.xfs --cloud ...' param, that would at the very least give us something to evangelize to users as these "why does my 3TB 128MB agsize fs suck?" reports continue to come in..
Yes!
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I.e., I'm wondering if the size is that large because you really need the space for data, or rather as a workaround where it could otherwise be formatted much smaller if the deployment side could grow it out to well beyond 37GB without having issues.
Quoted 8 lines
Quoted 5 lines
It depends on the workload -- vanilla OL images only give you a rootfs and assume that you'll create a separate higher iops block volume for actual data storage. $database automates provisioning and autotuning the separate block device for you.
Right.. so I assume these filesystems can be sized appropriately from the start.
<nod>
Quoted 23 lines
Quoted 5 lines
Another problem is that we don't necessarily know how many CPUs the system will have. I've seen a system with ~200 cores all pounding on a fs with four AGs (oops! lots of AGF contention!) or 10000 AGs (all cpus pounding on the radix tree).
Indeed.
Quoted 14 lines
IOWs, the deploy image could be a lot smaller, but apparently they're 48GB in size. My guess is that these are sparse images, since the billable size is about a tenth of that.
Quoted 4 lines
With a 37GB 4xAG fs, that's still closer to ~9GB AGs, which doesn't seem all that bad from a grow perspective. IIRC the insane filesystems I've seen tend to have much smaller (< 1GB maybe?) AG sizes. Have you seen grow issues with these ~9GB AG size fs'?
Not on our homegrown images. The problems crop up when people try to import images from outside $cloud and generate those minified images with 1G AGs. Unfortunately some of those people aren't even outside the org, but I'll get to them... ;)
Quoted 6 lines
Quoted 1 line
Ok. This is the part that is still somewhat mysterious to me.
Yeah, the outside image thing is mysterious to us as well. Customers make those images, so we need sane(ish) defaults, or some serious evangelization of the --cloud option.
Hm yeah. One of the things I've discussed with Eric a bit is how all over the place and unknown the use cases seem to be. I actually think there's benefit in just doing something simple with an abstracted mkfs param because if whatever initial heuristic we come up with isn't the best, it won't be any worse than current default behavior and we at least start to get some feedback and can iterate on it.
Quoted 13 lines
Quoted 5 lines
Given what we know about your internal images and how they are sized and whatnot, and expanding on the whole mkfs.xfs --cloud thought a bit, I'm wondering if we could just use something like that to enforce a minimum image size as well.
Quoted 7 lines
IOW, suppose 'mkfs.xfs --cloud ...' imposed a 20GB minimum file size and 10GB minimum AG size, documented that this is required for sane growability, and explained that the additional metadata over a 3GB image or whatever is mostly noise given sparse file image formats... do we think that would be sufficient to mostly avoid the problem? Would users adopt it if beat over the head enough with it when reporting problems with these ridiculous agcount filesystems? Hm?
Quoted 3 lines
I'd argue for a min sparse cloud image file size of 32G, and maybe even the one-and-runt AG setup above. But otherwise this sounds ok to me.
Sure, I've no strong opinion on the size limit.. Another thing Eric brought up that I'm not sure about with the whole "min file size" idea is whether any cloud environments would have issues with larger images because sparse files aren't handled efficiently enough by their infra, for whatever reason.
That seems like something that's easy enough to address with compression and such, but practicality doesn't always line up with theory and if that's not something that's tied well enough into the cloudy management infrastructure such that admins just don't use it, then that kind of defeats the purpose. I'm really not sure either way.. :/
Anyways, to be continued.. I'm curious on any thoughts you and others might have on the prototype.
Brian
Quoted 164 lines
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 31 lines
Quoted 23 lines
Quoted 6 lines
IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB.
This is my understanding as well.
Quoted 2 lines
I think this is worth prototyping a bit more. Do you?
I do as well..
Eric and I had a random discussion about this the other week and one of the concerns he brought up is the risk that some deployment might grow and still end up on a single AG fs because maybe the deployment just didn't provide sufficient storage as expected for the image. This was part of the reason I brought up the whole image mode feature bit in the expandfs discussion, because we could always do something to enforce that fs functionality is hobbled until sufficiently grown to at least 2xAGs and thus clear the image mode feature bit.
Quoted 7 lines
That said, another thing I played around a bit with after posting this was the ability to just grow the AG size at runtime. I.e., for a single AG fs, it's fairly easy to just change the AG size at growfs time. I currently have that prototyped by using a separate transaction to avoid some verifier failure madness I didn't want to sift through to prove the concept, but this could also be prototyped to exist as a separate, optional
GROWFS_AGSIZEioctl without much additional fuss.Quoted 4 lines
What if we rev the growfsdata ioctl to allow the user to specify an agsize? Userspace (i.e.
xfs_growfs) can then try to set the AG size, or if that fails, fall back to the current growfsdata.Either way seems reasonable. I think we're mostly talking about the same thing. For a functional prototype I'd be more concerned with just showing whether it works and then if so, whether it's useful/flexible enough for the intended use cases.
Quoted 6 lines
Quoted 2 lines
Agreed!
Ok. It's on my todo list to clean up what I've hacked up wrt the whole grow agsize idea enough so it can be at least put on the list for discussion/posterity.. thanks.
NP.
--D
Quoted 111 lines
Brian
Quoted 107 lines
Quoted 106 lines
Brian
Quoted 102 lines
Quoted 9 lines
So for example, suppose that we just created an image file fs with something like "mkfs.xfs -d agcount=1 -lsize=64m <file>," where the superblock AG size is not actually larger than the single AG. Then we update
xfs_growfsto look at the current block device size, make a decision on an ideal AG size from that, and run anioctl(GROWFS_AGSIZE, agsize)followed by the typicalioctl(GROWFS_DATA, bdev_size). TheGROWFS_AGSIZEcan either update the sb ag size as appropriate before the grow, or just fail and we fall back to default growfs behavior.Quoted 8 lines
That would allow a bit more dynamic runtime behavior and maybe ensure we end up with a more typical 4xAG fs in most cases. Dave had pointed out that there might be concerns around AG sizing logic, but this would exist in userspace and could be lifted straight from mkfs if need be. TBH, on my first look it looked like it was mostly alignment (i.e. stripe unit, etc.) related stuff, so I'd probably just punt and let
GROWFS_AGSIZEfail if any such unsupported fields are set. Worst case we just fall back to existing behavior.I wouldn't have a problem with hoisting that to libxfs and sharing with the kernel.
Quoted 5 lines
Thoughts on that? If that sounds interesting enough I can follow up with a v2 prototype along those lines. Appreciate the discussion.
Brian
Quoted 8 lines
P.S. Semi-random thought after writing up the above.. it occurred to me that we already have a provisional shrink mechanism, so in theory perhaps we could also do something like "mkfs.xfs --image-mode <file>" where we just hardcoded agcount=1,agsize=1TB, and then let userspace grow shrink to the appropriate agsize in the delta < 0 && agsize > fssize case and then grow out normally from there. Maybe that's just another way of doing the same thing, but something else to think about at least..<shrug> I'd rather do the ag resizing explicitly than implied if someone says to shrink (or grow) and the agcount is zero and magic compat flag is set.
--D
Quoted 60 lines
Quoted 58 lines
--D
Quoted 54 lines
Brian
[1] https://lore.kernel.org/linux-xfs/[email protected]/
[2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/mkfs/xfs_mkfs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)mkfs/xfs_mkfs.c +5
-6@@ -325,8 +325,7 @@ static struct opt_params dopts = { }, .subopt_params = { { .index = D_AGCOUNT, - .conflicts = { { &dopts, D_AGSIZE }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .minval = 1, .maxval = XFS_MAX_AGNUMBER,@@ -368,8 +367,7 @@ static struct opt_params dopts = { .defaultval = SUBOPT_NEEDS_VAL, }, { .index = D_AGSIZE, - .conflicts = { { &dopts, D_AGCOUNT }, - { &dopts, D_CONCURRENCY }, + .conflicts = { { &dopts, D_CONCURRENCY }, { NULL, LAST_CONFLICT } }, .convert = true, .minval = XFS_AG_MIN_BYTES,@@ -1233,7 +1231,7 @@ validate_ag_geometry(usage (); } -if (agsize > dblocks) { +if (agsize > dblocks && agcount !=1 ) {fprintf (stderr,_ ("agsize (%lld blocks) too big, data area is %lld blocks\n" ), (long long )agsize, (long long )dblocks);@@ -2703,7 +2701,8 @@ validate_supported(* Filesystems should not have fewer than two AGs, because we need to * have redundant superblocks. */ -if (mp->m_sb.sb_agcount <2 ) { +if (mp->m_sb.sb_agcount <2 && + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) {fprintf (stderr,_ ("Filesystem must have at least 2 superblocks for redundancy!\n" ));usage ();Signature · 1 line
2.45.0