Skip to contentSkip to conversation

xfs: set max_agbno to allow sparse alloc of last full inode chunk

PATCH Superseded Brian Foster 2 participants 6 messages Jan 8 – Jan 9 1 message not in the archive XFS filesystem

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: set max_agbno to allow sparse alloc of last full inode chunk

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Note that this can result in filesystem shutdown via dirty trans cancel on stable kernels prior to commit 9eb775968b68 ("xfs: walk all AGs if TRYLOCK passed to xfs_alloc_vextent_iterate_ags") because the tail AG selection by the allocator sets t_highest_agno on the transaction. If the inode allocator spins around and finds an inode chunk with free inodes in an earlier AG, the subsequent dir name creation path may still fail to allocate due to the AG restriction and cancel.

To avoid this problem, update the max_agbno calculation to the agbno prior to the last chunk aligned agbno in the AG. This is not necessarily the last valid allocation target for a sparse chunk, but since inode chunks (i.e. records) are chunk aligned and sparse allocs are cluster sized/aligned, this allows the sb_spino_align alignment restriction to take over and round down the max effective agbno to within the last valid inode chunk in the AG.

Note that even though the allocator improvements in the aforementioned commit seem to avoid this particular dirty trans cancel situation, the max_agbno logic improvement still applies as we should be able to allocate from an AG that has been appropriately selected. The more important target for this patch however are older/stable kernels prior to this allocator rework/improvement.

Fixes 56d1115c9bc7 ("xfs: allocate sparse inode chunks on full chunk allocation failure")
---
 fs/xfs/libxfs/xfs_ialloc.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ialloc.c +6 -5
index d97295eaebe6..c19d6d713780 100644
@@ -848,15 +848,16 @@ xfs_ialloc_ag_alloc(
 		 * invalid inode records, such as records that start at agbno 0
 		 * or extend beyond the AG.
 		 *
-		 * Set min agbno to the first aligned, non-zero agbno and max to
-		 * the last aligned agbno that is at least one full chunk from
-		 * the end of the AG.
+		 * Set min agbno to the first chunk aligned, non-zero agbno and
+		 * max to one less than the last chunk aligned agbno from the
+		 * end of the AG. We subtract 1 from max so that the cluster
+		 * allocation alignment takes over and allows allocation within
+		 * the last full inode chunk in the AG.
 		 */
 		args.min_agbno = args.mp->m_sb.sb_inoalignmt;
 		args.max_agbno = round_down(xfs_ag_block_count(args.mp,
 							pag_agno(pag)),
-					    args.mp->m_sb.sb_inoalignmt) -
-				 igeo->ialloc_blks;
+					    args.mp->m_sb.sb_inoalignmt) - 1;
 
 		error = xfs_alloc_vextent_near_bno(&args,
 				xfs_agbno_to_fsb(pag,
Signature · 1 line
2.52.0
#1 lore raw
Quoted 31 lines · Brian Foster

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Note that this can result in filesystem shutdown via dirty trans cancel on stable kernels prior to commit 9eb775968b68 ("xfs: walk all AGs if TRYLOCK passed to xfs_alloc_vextent_iterate_ags") because the tail AG selection by the allocator sets t_highest_agno on the transaction. If the inode allocator spins around and finds an inode chunk with free inodes in an earlier AG, the subsequent dir name creation path may still fail to allocate due to the AG restriction and cancel.

To avoid this problem, update the max_agbno calculation to the agbno prior to the last chunk aligned agbno in the AG. This is not necessarily the last valid allocation target for a sparse chunk, but since inode chunks (i.e. records) are chunk aligned and sparse allocs are cluster sized/aligned, this allows the sb_spino_align alignment restriction to take over and round down the max effective agbno to within the last valid inode chunk in the AG.

Quoted 6 lines

Note that even though the allocator improvements in the aforementioned commit seem to avoid this particular dirty trans cancel situation, the max_agbno logic improvement still applies as we should be able to allocate from an AG that has been appropriately selected. The more important target for this patch however are older/stable kernels prior to this allocator rework/improvement.

<nod> It makes sense to me that we ought to be able to examine space out to the final(ish) agbno of the runt AG.

Question for you: There are 16 holemask bits for 64 inodes per inobt record, or in other words the new allocation has to be aligned at least to the number of blocks needed to write 4 inodes. I /think/ sb_spino_align reflects that, right?

Quoted 1 line
Fixes 56d1115c9bc7 ("xfs: allocate sparse inode chunks on full chunk allocation failure")
Cc <[email protected]> # v4.2

--D

Quoted 35 lines
---
 fs/xfs/libxfs/xfs_ialloc.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ialloc.c +6 -5
index d97295eaebe6..c19d6d713780 100644
@@ -848,15 +848,16 @@ xfs_ialloc_ag_alloc(
 		 * invalid inode records, such as records that start at agbno 0
 		 * or extend beyond the AG.
 		 *
-		 * Set min agbno to the first aligned, non-zero agbno and max to
-		 * the last aligned agbno that is at least one full chunk from
-		 * the end of the AG.
+		 * Set min agbno to the first chunk aligned, non-zero agbno and
+		 * max to one less than the last chunk aligned agbno from the
+		 * end of the AG. We subtract 1 from max so that the cluster
+		 * allocation alignment takes over and allows allocation within
+		 * the last full inode chunk in the AG.
 		 */
 		args.min_agbno = args.mp->m_sb.sb_inoalignmt;
 		args.max_agbno = round_down(xfs_ag_block_count(args.mp,
 							pag_agno(pag)),
-					    args.mp->m_sb.sb_inoalignmt) -
-				 igeo->ialloc_blks;
+					    args.mp->m_sb.sb_inoalignmt) - 1;
 
 		error = xfs_alloc_vextent_near_bno(&args,
 				xfs_agbno_to_fsb(pag,
Signature · 1 line
2.52.0
#2 lore raw
Quoted 39 lines · Darrick J. Wong
Quoted 31 lines · Brian Foster

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Note that this can result in filesystem shutdown via dirty trans cancel on stable kernels prior to commit 9eb775968b68 ("xfs: walk all AGs if TRYLOCK passed to xfs_alloc_vextent_iterate_ags") because the tail AG selection by the allocator sets t_highest_agno on the transaction. If the inode allocator spins around and finds an inode chunk with free inodes in an earlier AG, the subsequent dir name creation path may still fail to allocate due to the AG restriction and cancel.

To avoid this problem, update the max_agbno calculation to the agbno prior to the last chunk aligned agbno in the AG. This is not necessarily the last valid allocation target for a sparse chunk, but since inode chunks (i.e. records) are chunk aligned and sparse allocs are cluster sized/aligned, this allows the sb_spino_align alignment restriction to take over and round down the max effective agbno to within the last valid inode chunk in the AG.

Quoted 6 lines

Note that even though the allocator improvements in the aforementioned commit seem to avoid this particular dirty trans cancel situation, the max_agbno logic improvement still applies as we should be able to allocate from an AG that has been appropriately selected. The more important target for this patch however are older/stable kernels prior to this allocator rework/improvement.

Quoted 8 lines

<nod> It makes sense to me that we ought to be able to examine space out to the final(ish) agbno of the runt AG.

Question for you: There are 16 holemask bits for 64 inodes per inobt record, or in other words the new allocation has to be aligned at least to the number of blocks needed to write 4 inodes. I /think/ sb_spino_align reflects that, right?

Pretty much.. If I recall all the details correctly, the holemask bit per 4-inode ratio was more of a data structure thing. That was just based on how much space we had in the standard inode chunk record freecount field to repurpose to track "holes" in an inode chunk.

The alignment rules had to change for higher level design raisins, because if we allocated some sparse chunk out of fragmented free space we need a consistent way to map it back to an inode record without causing conflicts across multiple inode records (i.e. accidental record overlap or whatever else). So therefore when sparse inodes are enabled, at mkfs time we change inode chunk alignment from cluster size to full inode chunk size, and set sparse chunk alignment to the cluster size.

This creates an inherent mapping for a sparse inode chunk to an inode record because the cluster aligned sparse chunk always maps to whatever chunk aligned record covers it (so we know whether to allocate a new inode record or use one that might already be sparse based on the sparse alloc, etc.).

Quoted 4 lines
Quoted 1 line
Fixes 56d1115c9bc7 ("xfs: allocate sparse inode chunks on full chunk allocation failure")
Cc <[email protected]> # v4.2

Thanks. Do you want me to repost with that or shall the maintainer handle it? ;)

Brian

Quoted 38 lines

--D

Quoted 35 lines
---
 fs/xfs/libxfs/xfs_ialloc.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ialloc.c +6 -5
index d97295eaebe6..c19d6d713780 100644
@@ -848,15 +848,16 @@ xfs_ialloc_ag_alloc(
 		 * invalid inode records, such as records that start at agbno 0
 		 * or extend beyond the AG.
 		 *
-		 * Set min agbno to the first aligned, non-zero agbno and max to
-		 * the last aligned agbno that is at least one full chunk from
-		 * the end of the AG.
+		 * Set min agbno to the first chunk aligned, non-zero agbno and
+		 * max to one less than the last chunk aligned agbno from the
+		 * end of the AG. We subtract 1 from max so that the cluster
+		 * allocation alignment takes over and allows allocation within
+		 * the last full inode chunk in the AG.
 		 */
 		args.min_agbno = args.mp->m_sb.sb_inoalignmt;
 		args.max_agbno = round_down(xfs_ag_block_count(args.mp,
 							pag_agno(pag)),
-					    args.mp->m_sb.sb_inoalignmt) -
-				 igeo->ialloc_blks;
+					    args.mp->m_sb.sb_inoalignmt) - 1;
 
 		error = xfs_alloc_vextent_near_bno(&args,
 				xfs_agbno_to_fsb(pag,
Signature · 1 line
2.52.0
#6 lore raw

Re: [External] : [PATCH] xfs: set max_agbno to allow sparse alloc of last full inode chunk

In reply to a message not in the archive

Adding linux-xfs back to cc.

Quoted 16 lines · Mark Tinguely
Quoted 5 lines · Brian Foster

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

Quoted 9 lines

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Quoted 3 lines

With the same logic, wouldn't the 4 block cluster at agbno 2056 also be a valid sparse inode cluster?

Nope.. the problem there is that 4 block cluster would map to an inode record at the same agbno 2056, but that record in the inobt would be invalid because there are only 7 blocks before the end of the 2063 block runt AG (full inode records are 8 blocks in this example). So IIRC the metadata verifiers will complain about this and consider it corruption and whatnot.

That was actually the issue fixed by Dave's more recent commit 13325333582d ("xfs: fix sparse inode limits on runt AG") in this same area. The subtlety here is that the calculation was off in this regard from the start, but it never mattered as such because it wasn't effective in this small runt AG case in the first place. So that fix made the min/max agbno bounding logic effective, and with that in place this fell out more recently pointing out that the original calculation was a bit too conservative.

Brian

Quoted 4 lines

thanks,

Mark.

Reviewed-by
#3 lore raw
In reply to Brian Foster
Quoted 62 lines · Brian Foster
Quoted 39 lines · Darrick J. Wong
Quoted 31 lines · Brian Foster

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Note that this can result in filesystem shutdown via dirty trans cancel on stable kernels prior to commit 9eb775968b68 ("xfs: walk all AGs if TRYLOCK passed to xfs_alloc_vextent_iterate_ags") because the tail AG selection by the allocator sets t_highest_agno on the transaction. If the inode allocator spins around and finds an inode chunk with free inodes in an earlier AG, the subsequent dir name creation path may still fail to allocate due to the AG restriction and cancel.

To avoid this problem, update the max_agbno calculation to the agbno prior to the last chunk aligned agbno in the AG. This is not necessarily the last valid allocation target for a sparse chunk, but since inode chunks (i.e. records) are chunk aligned and sparse allocs are cluster sized/aligned, this allows the sb_spino_align alignment restriction to take over and round down the max effective agbno to within the last valid inode chunk in the AG.

Quoted 6 lines

Note that even though the allocator improvements in the aforementioned commit seem to avoid this particular dirty trans cancel situation, the max_agbno logic improvement still applies as we should be able to allocate from an AG that has been appropriately selected. The more important target for this patch however are older/stable kernels prior to this allocator rework/improvement.

Quoted 8 lines

<nod> It makes sense to me that we ought to be able to examine space out to the final(ish) agbno of the runt AG.

Question for you: There are 16 holemask bits for 64 inodes per inobt record, or in other words the new allocation has to be aligned at least to the number of blocks needed to write 4 inodes. I /think/ sb_spino_align reflects that, right?

Pretty much.. If I recall all the details correctly, the holemask bit per 4-inode ratio was more of a data structure thing. That was just based on how much space we had in the standard inode chunk record freecount field to repurpose to track "holes" in an inode chunk.

The alignment rules had to change for higher level design raisins, because if we allocated some sparse chunk out of fragmented free space we need a consistent way to map it back to an inode record without causing conflicts across multiple inode records (i.e. accidental record overlap or whatever else). So therefore when sparse inodes are enabled, at mkfs time we change inode chunk alignment from cluster size to full inode chunk size, and set sparse chunk alignment to the cluster size.

Quoted 5 lines

This creates an inherent mapping for a sparse inode chunk to an inode record because the cluster aligned sparse chunk always maps to whatever chunk aligned record covers it (so we know whether to allocate a new inode record or use one that might already be sparse based on the sparse alloc, etc.).

<nod> Ok, that's what I was thinking, so I'm glad I asked. :)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 56d1115c9bc7 ("xfs: allocate sparse inode chunks on full chunk allocation failure")
Cc <[email protected]> # v4.2

Thanks. Do you want me to repost with that or shall the maintainer handle it? ;)

Well first things first ;)

If you want to repost with the cc and the rvb tag then please do that. If not, then Carlos, can you include both when you add this to for-next, please?

--D

Quoted 42 lines

Brian

Quoted 38 lines

--D

Quoted 35 lines
---
 fs/xfs/libxfs/xfs_ialloc.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ialloc.c +6 -5
index d97295eaebe6..c19d6d713780 100644
@@ -848,15 +848,16 @@ xfs_ialloc_ag_alloc(
 		 * invalid inode records, such as records that start at agbno 0
 		 * or extend beyond the AG.
 		 *
-		 * Set min agbno to the first aligned, non-zero agbno and max to
-		 * the last aligned agbno that is at least one full chunk from
-		 * the end of the AG.
+		 * Set min agbno to the first chunk aligned, non-zero agbno and
+		 * max to one less than the last chunk aligned agbno from the
+		 * end of the AG. We subtract 1 from max so that the cluster
+		 * allocation alignment takes over and allows allocation within
+		 * the last full inode chunk in the AG.
 		 */
 		args.min_agbno = args.mp->m_sb.sb_inoalignmt;
 		args.max_agbno = round_down(xfs_ag_block_count(args.mp,
 							pag_agno(pag)),
-					    args.mp->m_sb.sb_inoalignmt) -
-				 igeo->ialloc_blks;
+					    args.mp->m_sb.sb_inoalignmt) - 1;
 
 		error = xfs_alloc_vextent_near_bno(&args,
 				xfs_agbno_to_fsb(pag,
Signature · 1 line
2.52.0
#4 lore raw
Quoted 79 lines · Darrick J. Wong
Quoted 62 lines · Brian Foster
Quoted 39 lines · Darrick J. Wong
Quoted 31 lines · Brian Foster

Sparse inode cluster allocation sets min/max agbno values to avoid allocating an inode cluster that might map to an invalid inode chunk. For example, we can't have an inode record mapped to agbno 0 or that extends past the end of a runt AG of misaligned size.

The initial calculation of max_agbno is unnecessarily conservative, however. This has triggered a corner case allocation failure where a small runt AG (i.e. 2063 blocks) is mostly full save for an extent to the EOFS boundary: [2050,13]. max_agbno is set to 2048 in this case, which happens to be the offset of the last possible valid inode chunk in the AG. In practice, we should be able to allocate the 4-block cluster at agbno 2052 to map to the parent inode record at agbno 2048, but the max_agbno value precludes it.

Note that this can result in filesystem shutdown via dirty trans cancel on stable kernels prior to commit 9eb775968b68 ("xfs: walk all AGs if TRYLOCK passed to xfs_alloc_vextent_iterate_ags") because the tail AG selection by the allocator sets t_highest_agno on the transaction. If the inode allocator spins around and finds an inode chunk with free inodes in an earlier AG, the subsequent dir name creation path may still fail to allocate due to the AG restriction and cancel.

To avoid this problem, update the max_agbno calculation to the agbno prior to the last chunk aligned agbno in the AG. This is not necessarily the last valid allocation target for a sparse chunk, but since inode chunks (i.e. records) are chunk aligned and sparse allocs are cluster sized/aligned, this allows the sb_spino_align alignment restriction to take over and round down the max effective agbno to within the last valid inode chunk in the AG.

Quoted 6 lines

Note that even though the allocator improvements in the aforementioned commit seem to avoid this particular dirty trans cancel situation, the max_agbno logic improvement still applies as we should be able to allocate from an AG that has been appropriately selected. The more important target for this patch however are older/stable kernels prior to this allocator rework/improvement.

Quoted 8 lines

<nod> It makes sense to me that we ought to be able to examine space out to the final(ish) agbno of the runt AG.

Question for you: There are 16 holemask bits for 64 inodes per inobt record, or in other words the new allocation has to be aligned at least to the number of blocks needed to write 4 inodes. I /think/ sb_spino_align reflects that, right?

Pretty much.. If I recall all the details correctly, the holemask bit per 4-inode ratio was more of a data structure thing. That was just based on how much space we had in the standard inode chunk record freecount field to repurpose to track "holes" in an inode chunk.

The alignment rules had to change for higher level design raisins, because if we allocated some sparse chunk out of fragmented free space we need a consistent way to map it back to an inode record without causing conflicts across multiple inode records (i.e. accidental record overlap or whatever else). So therefore when sparse inodes are enabled, at mkfs time we change inode chunk alignment from cluster size to full inode chunk size, and set sparse chunk alignment to the cluster size.

Quoted 5 lines

This creates an inherent mapping for a sparse inode chunk to an inode record because the cluster aligned sparse chunk always maps to whatever chunk aligned record covers it (so we know whether to allocate a new inode record or use one that might already be sparse based on the sparse alloc, etc.).

<nod> Ok, that's what I was thinking, so I'm glad I asked. :)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 56d1115c9bc7 ("xfs: allocate sparse inode chunks on full chunk allocation failure")
Cc <[email protected]> # v4.2

Thanks. Do you want me to repost with that or shall the maintainer handle it? ;)

Quoted 8 lines

Well first things first ;)

If you want to repost with the cc and the rvb tag then please do that. If not, then Carlos, can you include both when you add this to for-next, please?

Thanks! No problem.. I'll spin a v2 with all of this in a sec..

Brian

Quoted 46 lines

--D

Quoted 42 lines

Brian

Quoted 38 lines

--D

Quoted 35 lines
---
 fs/xfs/libxfs/xfs_ialloc.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_ialloc.c +6 -5
index d97295eaebe6..c19d6d713780 100644
@@ -848,15 +848,16 @@ xfs_ialloc_ag_alloc(
 		 * invalid inode records, such as records that start at agbno 0
 		 * or extend beyond the AG.
 		 *
-		 * Set min agbno to the first aligned, non-zero agbno and max to
-		 * the last aligned agbno that is at least one full chunk from
-		 * the end of the AG.
+		 * Set min agbno to the first chunk aligned, non-zero agbno and
+		 * max to one less than the last chunk aligned agbno from the
+		 * end of the AG. We subtract 1 from max so that the cluster
+		 * allocation alignment takes over and allows allocation within
+		 * the last full inode chunk in the AG.
 		 */
 		args.min_agbno = args.mp->m_sb.sb_inoalignmt;
 		args.max_agbno = round_down(xfs_ag_block_count(args.mp,
 							pag_agno(pag)),
-					    args.mp->m_sb.sb_inoalignmt) -
-				 igeo->ialloc_blks;
+					    args.mp->m_sb.sb_inoalignmt) - 1;
 
 		error = xfs_alloc_vextent_near_bno(&args,
 				xfs_agbno_to_fsb(pag,
Signature · 1 line
2.52.0