repair: fix AGFL rebuild problem

From: Dave Chinner <[email protected]>

Obviously hasn't been used for quite some time, so fix the build problems and make it useful again.

Signed-off-by Dave Chinner <[email protected]>

---
 repair/phase5.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

repair/phase5.c +3 -8

index e82eeb7..d6d3c6d 100644
@@ -362,10 +362,6 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	extent_tree_node_t	*ext_ptr;
 	int			level;
 #ifdef XR_BLD_FREE_TRACE
-	int			old_state;
-	int			state = XR_E_BAD_STATE;
-#endif
-#ifdef XR_BLD_FREE_TRACE
 	fprintf(stderr,
 		"in init_freespace_cursor, agno = %d\n", agno);
 #endif

@@ -1292,8 +1288,8 @@ build_agf_agfl(xfs_mount_t	*mp,
 	memset(agf, 0, mp->m_sb.sb_sectsize);
 
 #ifdef XR_BLD_FREE_TRACE
-	fprintf(stderr, "agf = 0x%x, agf_buf->b_un.b_addr = 0x%x\n",
-		(__psint_t) agf, (__psint_t) agf_buf->b_un.b_addr);
+	fprintf(stderr, "agf = 0x%p, agf_buf->b_addr = 0x%p\n",
+		agf, agf_buf->b_addr);
 #endif
 
 	/*

@@ -1452,7 +1448,7 @@ build_agf_agfl(xfs_mount_t	*mp,
 	}
 
 #ifdef XR_BLD_FREE_TRACE
-	fprintf(stderr, "wrote agf for ag %u, error = %d\n", agno, error);
+	fprintf(stderr, "wrote agf for ag %u\n", agno);
 #endif
 }
 

@@ -1707,11 +1703,10 @@ phase5(xfs_mount_t *mp)
 #ifdef XR_BLD_FREE_TRACE
 	fprintf(stderr, "inobt level 1, maxrec = %d, minrec = %d\n",
 		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, 0),
-		xfs_inobt_maxrecs(mp->m_sb.sb_blocksize, 0) / 2

From: Dave Chinner <[email protected]>

Obviously hasn't been used for quite some time, so fix the build problems and make it useful again.

Signed-off-by Dave Chinner <[email protected]>

---

 repair/phase5.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

repair/phase5.c +6 -11

index e82eeb7..d6d3c6d 100644
@@ -362,10 +362,6 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	extent_tree_node_t	*ext_ptr;
 	int			level;
 #ifdef XR_BLD_FREE_TRACE
-	int			old_state;
-	int			state = XR_E_BAD_STATE;
-#endif
-#ifdef XR_BLD_FREE_TRACE
 	fprintf(stderr,
 		"in init_freespace_cursor, agno = %d\n", agno);
 #endif

@@ -1292,8 +1288,8 @@ build_agf_agfl(xfs_mount_t	*mp,
 	memset(agf, 0, mp->m_sb.sb_sectsize);
 
 #ifdef XR_BLD_FREE_TRACE
-	fprintf(stderr, "agf = 0x%x, agf_buf->b_un.b_addr = 0x%x\n",
-		(__psint_t) agf, (__psint_t) agf_buf->b_un.b_addr);
+	fprintf(stderr, "agf = 0x%p, agf_buf->b_addr = 0x%p\n",
+		agf, agf_buf->b_addr);
 #endif
 
 	/*

@@ -1452,7 +1448,7 @@ build_agf_agfl(xfs_mount_t	*mp,
 	}
 
 #ifdef XR_BLD_FREE_TRACE
-	fprintf(stderr, "wrote agf for ag %u, error = %d\n", agno, error);
+	fprintf(stderr, "wrote agf for ag %u\n", agno);
 #endif
 }
 

@@ -1707,11 +1703,10 @@ phase5(xfs_mount_t *mp)
 #ifdef XR_BLD_FREE_TRACE
 	fprintf(stderr, "inobt level 1, maxrec = %d, minrec = %d\n",
 		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, 0),
-		xfs_inobt_maxrecs(mp->m_sb.sb_blocksize, 0) / 2
-		);
+		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, 0) / 2);
 	fprintf(stderr, "inobt level 0 (leaf), maxrec = %d, minrec = %d\n",
-		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, xfs_inobt, 1),
-		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, xfs_inobt, 1) / 2);
+		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, 1),
+		xfs_inobt_maxrecs(mp, mp->m_sb.sb_blocksize, 1) / 2);
 	fprintf(stderr, "xr inobt level 0 (leaf), maxrec = %d\n",
 		XR_INOBT_BLOCK_MAXRECS(mp, 0));
 	fprintf(stderr, "xr inobt level 1 (int), maxrec = %d\n",

Signature · 6 lines

2.0.0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

From: Dave Chinner <[email protected]>

In phase 5 we rebuild the freelist btrees, the AGF and the AGFL from all the free space information we have gathered and resolved during pahses 3 and 4. If the freespace information is laid out just right, we end up having to allocate free space for the AGFL blocks.

If the size of the free space we allocate from is larger than the space we need, then we have to insert the remainder back into the freespace btree. For the by-size tree, this means we are likely to be removing a record from one leaf, and then inserting the remainder
- a smaller size - into another leaf.

The issue is that the leaf blocks to the left of the original leaf block we removed the extent record from are full and hence require a split to insert the new record. That, of course, requires a free block in the AGFL to allocate from, and now we have a chicken and egg situation: there are no free blocks in the AGFL because we are setting it up.

As a result, setting up the free list silently fails, leaving the freespace btrees in an inconsistent state and the AGFL in question empty. When the filesystem is next mounted, the first allocation from that AGF results in attempting to fix the AGFL, and it then does exactly the same thing as the repair code, fails to allocate a block during the split and fails. This results in an immediate shutdown because the transaction doing the allocation is dirty by this stage.

The fix for the problem is to make repair handle rebulding the btree differently. If we leave ispace for a couple of records in each btree leaf and node, there is never a need for a split to occur when initially setting up the AGFL. This results in repair doing the right thing, and hence the runtime problems after mount don't occur. Further, add error checking the the AGFL setup code and abort repair if we have a failure to correctly set up the AGFL so we catch this problem at repair time, not mount time...

Reported-by Barkley Vowk <[email protected]>

Signed-off-by Dave Chinner <[email protected]>

---
 repair/phase5.c | 44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)

repair/phase5.c +3 -0

index d6d3c6d..3d58936 100644
@@ -335,11 +335,22 @@ finish_cursor(bt_status_t *curs)
 }
 
 /*
+ * We need to leave some free records in the tree for the corner case of
+ * setting up the AGFL. This may require allocation of blocks, and as
+ * such can require insertion of new records into the tree (e.g. moving

repair/phase5.c +1 -6

  */
 #define XR_ALLOC_BLOCK_MAXRECS(mp, level) \
-			xfs_allocbt_maxrecs((mp), (mp)->m_sb.sb_blocksize, \
-						(level) == 0)
+	(xfs_allocbt_maxrecs((mp), (mp)->m_sb.sb_blocksize, (level) == 0) - 2)
 
 /*
  * this calculates a freespace cursor for an ag.

@@ -361,10 +372,6 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	bt_stat_level_t		*p_lptr;
 	extent_tree_node_t	*ext_ptr;
 	int			level;
-#ifdef XR_BLD_FREE_TRACE
-	fprintf(stderr,
-		"in init_freespace_cursor, agno = %d\n", agno);
-#endif
 
 	num_extents = *extents;
 	extents_used = 0;

repair/phase5.c +20 -3

+			lptr->num_blocks,
+			lptr->num_recs_pb,
+			lptr->modulo,
+			lptr->num_recs_tot);
+#endif
 	/*
 	 * if we need more levels, set them up.  # of records
 	 * per level is the # of blocks in the level below it

@@ -402,6 +416,14 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 			lptr->num_recs_pb = p_lptr->num_blocks
 					/ lptr->num_blocks;
 			lptr->num_recs_tot = p_lptr->num_blocks;
+#ifdef XR_BLD_FREE_TRACE
+			fprintf(stderr, "%s %d %d %d %d %d\n", __func__,
+					level,
+					lptr->num_blocks,
+					lptr->num_recs_pb,
+					lptr->modulo,
+					lptr->num_recs_tot);
+#endif
 		}
 	}
 

@@ -546,8 +568,7 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 				lptr = &btree_curs->level[level];
 				p_lptr = &btree_curs->level[level-1];
 				lptr->num_blocks = howmany(p_lptr->num_blocks,
-						XR_ALLOC_BLOCK_MAXRECS(mp,
-								level));
+					XR_ALLOC_BLOCK_MAXRECS(mp, level));
 				lptr->modulo = p_lptr->num_blocks
 						% lptr->num_blocks;
 				lptr->num_recs_pb = p_lptr->num_blocks

@@ -1434,6 +1455,7 @@ build_agf_agfl(xfs_mount_t	*mp,
 		xfs_alloc_arg_t	args;
 		xfs_trans_t	*tp;
 		struct xfs_trans_res tres = {0};
+		int		error;
 
 		memset(&args, 0, sizeof(args));
 		args.tp = tp = libxfs_trans_alloc(mp, 0);

@@ -1442,8 +1464,12 @@ build_agf_agfl(xfs_mount_t	*mp,
 		args.alignment = 1;
 		args.pag = xfs_perag_get(mp,agno);
 		libxfs_trans_reserve(tp, &tres, XFS_MIN_FREELIST(agf, mp), 0);
-		libxfs_alloc_fix_freelist(&args, 0);
+		error = libxfs_alloc_fix_freelist(&args, 0);
 		xfs_perag_put(args.pag);
+		if (error) {
+			do_error(_("failed to fix AGFL on AG %d, error %d\n"),
+					agno, error);
+		}
 		libxfs_trans_commit(tp, 0);
 	}
 

Signature · 6 lines

2.0.0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 29 lines · Dave Chinner

From: Dave Chinner <[email protected]>

In phase 5 we rebuild the freelist btrees, the AGF and the AGFL from all the free space information we have gathered and resolved during pahses 3 and 4. If the freespace information is laid out just right, we end up having to allocate free space for the AGFL blocks.

If the size of the free space we allocate from is larger than the space we need, then we have to insert the remainder back into the freespace btree. For the by-size tree, this means we are likely to be removing a record from one leaf, and then inserting the remainder
- a smaller size - into another leaf.

The issue is that the leaf blocks to the left of the original leaf block we removed the extent record from are full and hence require a split to insert the new record. That, of course, requires a free block in the AGFL to allocate from, and now we have a chicken and egg situation: there are no free blocks in the AGFL because we are setting it up.

As a result, setting up the free list silently fails, leaving the freespace btrees in an inconsistent state and the AGFL in question empty. When the filesystem is next mounted, the first allocation from that AGF results in attempting to fix the AGFL, and it then does exactly the same thing as the repair code, fails to allocate a block during the split and fails. This results in an immediate shutdown because the transaction doing the allocation is dirty by this stage.

Quoted 9 lines

The fix for the problem is to make repair handle rebulding the btree differently. If we leave ispace for a couple of records in each btree leaf and node, there is never a need for a split to occur when initially setting up the AGFL. This results in repair doing the right thing, and hence the runtime problems after mount don't occur. Further, add error checking the the AGFL setup code and abort repair if we have a failure to correctly set up the AGFL so we catch this problem at repair time, not mount time...

Interesting problem, thanks for the breakdown. I suppose the only interesting side effect is that the alloc btrees might require a bit more space after repair than before. I wonder if we need to take that into consideration here for certain cases. E.g., does this have ramifications for running repair on a clean, but completely full fs, or should that be generally handled by the existence of reserved blocks?

A couple minor nits below...

Quoted 13 lines

Reported-by Barkley Vowk <[email protected]>

Signed-off-by Dave Chinner <[email protected]>

---
 repair/phase5.c | 44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)

repair/phase5.c +0 -0

index d6d3c6d..3d58936 100644
@@ -335,11 +335,22 @@ finish_cursor(bt_status_t *curs)
 }
 

Quoted 4 lines

repair/phase5.c +3 -0

 /*
+ * We need to leave some free records in the tree for the corner case of
+ * setting up the AGFL. This may require allocation of blocks, and as
+ * such can require insertion of new records into the tree (e.g. moving

repair/phase5.c finish_cursor(bt_status_t *curs) · line 350

Quoted 10 lines

repair/phase5.c +9 -0

+ * a record in the by-count tree when a long extent is shortened). If we
+ * pack the records into the leaves with no slack space, this requires a
+ * leaf split to occur and a block to be allocated from the free list.
+ * If we don't have any blocks on the free list (because we are setting
+ * it up!), then we fail, and the filesystem will fail with the same
+ * failure at runtime. Hence leave a couple of records slack space in
+ * each block to allow immediate modification of the tree without
+ * requiring splits to be done.
+ *
  * XXX(hch): any reason we don't just look at mp->m_alloc_mxr?

I guess we could kill this comment now. ;)

repair/phase5.c calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, · line 396

Quoted 6 lines

repair/phase5.c +2 -0

@@ -385,6 +392,13 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	lptr->num_recs_tot = num_extents;
 	level = 1;
 
+#ifdef XR_BLD_FREE_TRACE
+	fprintf(stderr, "%s 0 %d %d %d %d\n", __func__,

What's the 0 for?

repair/phase5.c calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, · line 422

Quoted 8 lines

repair/phase5.c +4 -0

@@ -402,6 +416,14 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 			lptr->num_recs_pb = p_lptr->num_blocks
 					/ lptr->num_blocks;
 			lptr->num_recs_tot = p_lptr->num_blocks;
+#ifdef XR_BLD_FREE_TRACE
+			fprintf(stderr, "%s %d %d %d %d %d\n", __func__,
+					level,
+					lptr->num_blocks,

Quoted 40 lines · Brian Foster

Quoted 29 lines · Dave Chinner

From: Dave Chinner <[email protected]>

In phase 5 we rebuild the freelist btrees, the AGF and the AGFL from all the free space information we have gathered and resolved during pahses 3 and 4. If the freespace information is laid out just right, we end up having to allocate free space for the AGFL blocks.

If the size of the free space we allocate from is larger than the space we need, then we have to insert the remainder back into the freespace btree. For the by-size tree, this means we are likely to be removing a record from one leaf, and then inserting the remainder
- a smaller size - into another leaf.

The issue is that the leaf blocks to the left of the original leaf block we removed the extent record from are full and hence require a split to insert the new record. That, of course, requires a free block in the AGFL to allocate from, and now we have a chicken and egg situation: there are no free blocks in the AGFL because we are setting it up.

As a result, setting up the free list silently fails, leaving the freespace btrees in an inconsistent state and the AGFL in question empty. When the filesystem is next mounted, the first allocation from that AGF results in attempting to fix the AGFL, and it then does exactly the same thing as the repair code, fails to allocate a block during the split and fails. This results in an immediate shutdown because the transaction doing the allocation is dirty by this stage.

Quoted 9 lines

The fix for the problem is to make repair handle rebulding the btree differently. If we leave ispace for a couple of records in each btree leaf and node, there is never a need for a split to occur when initially setting up the AGFL. This results in repair doing the right thing, and hence the runtime problems after mount don't occur. Further, add error checking the the AGFL setup code and abort repair if we have a failure to correctly set up the AGFL so we catch this problem at repair time, not mount time...

Quoted 6 lines

Interesting problem, thanks for the breakdown. I suppose the only interesting side effect is that the alloc btrees might require a bit more space after repair than before. I wonder if we need to take that into consideration here for certain cases. E.g., does this have ramifications for running repair on a clean, but completely full fs, or should that be generally handled by the existence of reserved blocks?

In general, it shouldn't be an issue. It is extremely rare for a free space btree to be entirely compact - it currently only happens when repair runs, and the first insert at each leaf triggers splits and so after a short runtime will no longer be compact. Hence it is almost always going to be the case that the btree is smaller after repair than it was before....

I can see that it might cause an issue at maximally sized filesystems with worst case free space fragmentation, but given the runtime of repair of such a filesystem is probably measured in months and in hundreds of petabytes of RAM I'm not really concerned about that right now.

Quoted 16 lines

A couple minor nits below...

Quoted 13 lines

Reported-by Barkley Vowk <[email protected]>

Signed-off-by Dave Chinner <[email protected]>

---
 repair/phase5.c | 44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)

repair/phase5.c +0 -0

index d6d3c6d..3d58936 100644
@@ -335,11 +335,22 @@ finish_cursor(bt_status_t *curs)
 }
 

Quoted 16 lines

Quoted 4 lines

repair/phase5.c +3 -0

 /*
+ * We need to leave some free records in the tree for the corner case of
+ * setting up the AGFL. This may require allocation of blocks, and as
+ * such can require insertion of new records into the tree (e.g. moving

repair/phase5.c finish_cursor(bt_status_t *curs) · line 350

Quoted 10 lines

repair/phase5.c +9 -0

+ * a record in the by-count tree when a long extent is shortened). If we
+ * pack the records into the leaves with no slack space, this requires a
+ * leaf split to occur and a block to be allocated from the free list.
+ * If we don't have any blocks on the free list (because we are setting
+ * it up!), then we fail, and the filesystem will fail with the same
+ * failure at runtime. Hence leave a couple of records slack space in
+ * each block to allow immediate modification of the tree without
+ * requiring splits to be done.
+ *
  * XXX(hch): any reason we don't just look at mp->m_alloc_mxr?

I guess we could kill this comment now. ;)

Comment is still valid. I only fixed the bug - I didn't look into why we aren't using mp->m_alloc_mxr instead of calculating the max records value from first principles...

Quoted 8 lines

repair/phase5.c calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, · line 396

Quoted 6 lines

repair/phase5.c +2 -0

@@ -385,6 +392,13 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	lptr->num_recs_tot = num_extents;
 	level = 1;
 
+#ifdef XR_BLD_FREE_TRACE
+	fprintf(stderr, "%s 0 %d %d %d %d\n", __func__,

What's the 0 for?

The btree level. 0 == leaf....

....

Quoted 8 lines

repair/phase5.c calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, · line 422

Quoted 8 lines

repair/phase5.c +4 -0

@@ -402,6 +416,14 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 			lptr->num_recs_pb = p_lptr->num_blocks
 					/ lptr->num_blocks;
 			lptr->num_recs_tot = p_lptr->num_blocks;
+#ifdef XR_BLD_FREE_TRACE
+			fprintf(stderr, "%s %d %d %d %d %d\n", __func__,
+					level,
+					lptr->num_blocks,

And as you can see in this hunk as we set up the nodes the level is printed so we can see the config of each level as it is set up.

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi folks,

The following two patches fix a repair rebuild issue that was reported to me. xfs_repair would appear to succeed, but subsequent runs would report a freespace mismatch and then on mount trigger a corruption error on the first allocation from AG 0.

The problem as that the AGFL was not being correctly rebuilt due to a very particular freespace pattern in AG 0 which required free blocks in the AGFL to populate an empty AGFL. reapir silently ignored this error, so it triggered allocation corruption errors when the same AGFL fixup attempt failed on a mounted filesystem.

The fix is relatively straight forward, as is the fixes to the tracing code that I had to make before i could track down the issue.

-Dave.