Skip to contentSkip to conversation

xfs: fix recovery corruption on s390 w/ nrext64

PATCH In review Dave Chinner 2 participants 8 messages Nov 10, 2023 – Nov 17, 2023 XFS filesystem

lore
Share & tools
lore
#0 lore raw

cover xfs: fix recovery corruption on s390 w/ nrext64

Hi folks,

The following are two patches to fix the corruption bug that Zorro recently recently reported on big endian platforms when nrext64 is enabled.

It is caused by recovery zeroing a v2 inode field even on v3 inodes. Before nrext64, this wasn't an issue because the field in v3 inodes was just padding (i.e. zeroes) and so writing zero to part of it was never noticed. However, nrext64 uses this field for the 64 bit data fork extent count, so zeroing part of it becomes a problem.

The zeroing is done to the xfs_log_dinode, which is held in host format. Because of the layout of this field, on little endian machines the di_flush_iter field overlaps the high 16 bits of di_big_nextents, and so we didn't notice that recovery was overwriting the the high bits of the extent count because it's always going to be zero bytes (needs to go over 2^48 extents before we'd notice) in testing contexts.

However, on big endian machines, the di_flushiter field overlays the lower 16 bits of the di_big_extents field, and so zeroing di_flushiter essentially rounds it down to the nearest 64k. For typical testing contexts, that's effectively zeroing the extent count.

The fix is two parts. The first patch adds detection of this specific corruption to the xfs_dinode verifier, and adds a call to the verifier after recovery has recovered the inode. This catches corruptions at recovery time, rather than silently writing them back and not noticing them until the inode is re-read from disk at runtime.

The second patch fixes the recovery bug. It simply moves the resetting of the di_flushiter field into the v2 inode specific flushiter handling branch. That's always been wrong (i.e. since v3 inodes were introduced) but it hasn't mattered until now, so it marked as fixing the introduction of the nrext64 feature....

-Dave.

#1 lore raw

PATCH 1/2 xfs: inode recovery does not validate the recovered inode

From: Dave Chinner <[email protected]>

Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.

The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.

---
 fs/xfs/libxfs/xfs_inode_buf.c   |  3 +++
 fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_inode_buf.c +3 -0
index a35781577cad..0f970a0b3382 100644
@@ -508,6 +508,9 @@ xfs_dinode_verify(
 	if (mode && nextents + naextents > nblocks)
 		return __this_address;
 
+	if (nextents + naextents == 0 && nblocks != 0)
+		return __this_address;
+
 	if (S_ISDIR(mode) && nextents > mp->m_dir_geo->max_extents)
 		return __this_address;
 
fs/xfs/xfs_inode_item_recover.c +13 -1
index 6b09e2bf2d74..f4c31c2b60d5 100644
@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(
 	struct xfs_log_dinode		*ldip;
 	uint				isize;
 	int				need_free = 0;
+	xfs_failaddr_t			fa;
 
 	if (item->ri_buf[0].i_len == sizeof(struct xfs_inode_log_format)) {
 		in_f = item->ri_buf[0].i_addr;
@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2(
 	    (dip->di_mode != 0))
 		error = xfs_recover_inode_owner_change(mp, dip, in_f,
 						       buffer_list);
-	/* re-generate the checksum. */
+	/* re-generate the checksum and validate the recovered inode. */
 	xfs_dinode_calc_crc(log->l_mp, dip);
+	fa = xfs_dinode_verify(log->l_mp, in_f->ilf_ino, dip);
+	if (fa) {
+		XFS_CORRUPTION_ERROR(
+			"Bad dinode after recovery",
+				XFS_ERRLEVEL_LOW, mp, dip, sizeof(*dip));
+		xfs_alert(mp,
+			"Metadata corruption detected at %pS, inode 0x%llx",
+			fa, in_f->ilf_ino);
+		error = -EFSCORRUPTED;
+		goto out_release;
+	}
 
 	ASSERT(bp->b_mount == mp);
 	bp->b_flags |= _XBF_LOGRECOVERY;
Signature · 1 line
2.42.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode

Quoted 42 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.

The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.

---
 fs/xfs/libxfs/xfs_inode_buf.c   |  3 +++
 fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_inode_buf.c +3 -0
index a35781577cad..0f970a0b3382 100644
@@ -508,6 +508,9 @@ xfs_dinode_verify(
 	if (mode && nextents + naextents > nblocks)
 		return __this_address;
 
+	if (nextents + naextents == 0 && nblocks != 0)
+		return __this_address;
+
 	if (S_ISDIR(mode) && nextents > mp->m_dir_geo->max_extents)
 		return __this_address;
 
fs/xfs/xfs_inode_item_recover.c +1 -0
index 6b09e2bf2d74..f4c31c2b60d5 100644
@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(
 	struct xfs_log_dinode		*ldip;
 	uint				isize;
 	int				need_free = 0;
+	xfs_failaddr_t			fa;
 
 	if (item->ri_buf[0].i_len == sizeof(struct xfs_inode_log_format)) {
 		in_f = item->ri_buf[0].i_addr;

fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536

Quoted 8 lines
fs/xfs/xfs_inode_item_recover.c +3 -1
@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2(
 	    (dip->di_mode != 0))
 		error = xfs_recover_inode_owner_change(mp, dip, in_f,
 						       buffer_list);
-	/* re-generate the checksum. */
+	/* re-generate the checksum and validate the recovered inode. */
 	xfs_dinode_calc_crc(log->l_mp, dip);
+	fa = xfs_dinode_verify(log->l_mp, in_f->ilf_ino, dip);
+	if (fa) {

Does xlog_recover_dquot_commit_pass2 need to call xfs_dquot_verify as well?

This patch looks good though,

--D

Quoted 15 lines
fs/xfs/xfs_inode_item_recover.c +9 -0
+		XFS_CORRUPTION_ERROR(
+			"Bad dinode after recovery",
+				XFS_ERRLEVEL_LOW, mp, dip, sizeof(*dip));
+		xfs_alert(mp,
+			"Metadata corruption detected at %pS, inode 0x%llx",
+			fa, in_f->ilf_ino);
+		error = -EFSCORRUPTED;
+		goto out_release;
+	}
 
 	ASSERT(bp->b_mount == mp);
 	bp->b_flags |= _XBF_LOGRECOVERY;
Signature · 1 line
2.42.0
#3 lore raw

Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode

Quoted 40 lines · Darrick J. Wong
Quoted 39 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.

The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.

---
 fs/xfs/libxfs/xfs_inode_buf.c   |  3 +++
 fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_inode_buf.c +3 -0
index a35781577cad..0f970a0b3382 100644
@@ -508,6 +508,9 @@ xfs_dinode_verify(
 	if (mode && nextents + naextents > nblocks)
 		return __this_address;
 
+	if (nextents + naextents == 0 && nblocks != 0)
+		return __this_address;
+
 	if (S_ISDIR(mode) && nextents > mp->m_dir_geo->max_extents)
 		return __this_address;
 
fs/xfs/xfs_inode_item_recover.c +1 -0
index 6b09e2bf2d74..f4c31c2b60d5 100644
@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(
 	struct xfs_log_dinode		*ldip;
 	uint				isize;
 	int				need_free = 0;
+	xfs_failaddr_t			fa;
 
Quoted 14 lines

fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536

Quoted 11 lines
fs/xfs/xfs_inode_item_recover.c +3 -1
 	if (item->ri_buf[0].i_len == sizeof(struct xfs_inode_log_format)) {
 		in_f = item->ri_buf[0].i_addr;
@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2(
 	    (dip->di_mode != 0))
 		error = xfs_recover_inode_owner_change(mp, dip, in_f,
 						       buffer_list);
-	/* re-generate the checksum. */
+	/* re-generate the checksum and validate the recovered inode. */
 	xfs_dinode_calc_crc(log->l_mp, dip);
+	fa = xfs_dinode_verify(log->l_mp, in_f->ilf_ino, dip);
+	if (fa) {

Does xlog_recover_dquot_commit_pass2 need to call xfs_dquot_verify as well?

Maybe - I haven't looked closely at that, and it depends what the dquot buffer verifier does. If it's similar to the inode cluster buffer verifier (i.e. only checks for dquots, doesn't verify the dquots) then it should do the same thing. I don't have time to do this right now because I'm OOO for the next week, so maybe you could check this and send a patch for it?

Quoted 2 lines

This patch looks good though,

Thanks!

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw

Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode

Quoted 56 lines · Dave Chinner
Quoted 40 lines · Darrick J. Wong
Quoted 39 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.

The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.

---
 fs/xfs/libxfs/xfs_inode_buf.c   |  3 +++
 fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_inode_buf.c +3 -0
index a35781577cad..0f970a0b3382 100644
@@ -508,6 +508,9 @@ xfs_dinode_verify(
 	if (mode && nextents + naextents > nblocks)
 		return __this_address;
 
+	if (nextents + naextents == 0 && nblocks != 0)
+		return __this_address;
+
 	if (S_ISDIR(mode) && nextents > mp->m_dir_geo->max_extents)
 		return __this_address;
 
fs/xfs/xfs_inode_item_recover.c +1 -0
index 6b09e2bf2d74..f4c31c2b60d5 100644
@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(
 	struct xfs_log_dinode		*ldip;
 	uint				isize;
 	int				need_free = 0;
+	xfs_failaddr_t			fa;
 
Quoted 14 lines

fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536

Quoted 11 lines
fs/xfs/xfs_inode_item_recover.c +3 -1
 	if (item->ri_buf[0].i_len == sizeof(struct xfs_inode_log_format)) {
 		in_f = item->ri_buf[0].i_addr;
@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2(
 	    (dip->di_mode != 0))
 		error = xfs_recover_inode_owner_change(mp, dip, in_f,
 						       buffer_list);
-	/* re-generate the checksum. */
+	/* re-generate the checksum and validate the recovered inode. */
 	xfs_dinode_calc_crc(log->l_mp, dip);
+	fa = xfs_dinode_verify(log->l_mp, in_f->ilf_ino, dip);
+	if (fa) {

Does xlog_recover_dquot_commit_pass2 need to call xfs_dquot_verify as well?

Quoted 6 lines

Maybe - I haven't looked closely at that, and it depends what the dquot buffer verifier does. If it's similar to the inode cluster buffer verifier (i.e. only checks for dquots, doesn't verify the dquots) then it should do the same thing. I don't have time to do this right now because I'm OOO for the next week, so maybe you could check this and send a patch for it?

I tossed on a patch to do this and after a couple of days of generic/388 and generic/475 spinning I haven't noticed any failures.

--D

Quoted 9 lines
Quoted 2 lines

This patch looks good though,

Thanks!

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw

PATCH 2/2 xfs: recovery should not clear di_flushiter unconditionally

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Because on v3 inodes, di_flushiter doesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.

This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.

This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing di_flushiter on v2 inodes during recovery.

Fixes 9b7d16e34bbe ("xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers")
---
 fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
fs/xfs/xfs_inode_item_recover.c +17 -15
index f4c31c2b60d5..dbdab4ce7c44 100644
@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(
 	 * superblock flag to determine whether we need to look at di_flushiter
 	 * to skip replay when the on disk inode is newer than the log one
 	 */
-	if (!xfs_has_v3inodes(mp) &&
-	    ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
-		/*
-		 * Deal with the wrap case, DI_MAX_FLUSH is less
-		 * than smaller numbers
-		 */
-		if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
-		    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
-			/* do nothing */
-		} else {
-			trace_xfs_log_recover_inode_skip(log, in_f);
-			error = 0;
-			goto out_release;
+	if (!xfs_has_v3inodes(mp)) {
+		if (ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
+			/*
+			 * Deal with the wrap case, DI_MAX_FLUSH is less
+			 * than smaller numbers
+			 */
+			if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
+			    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
+				/* do nothing */
+			} else {
+				trace_xfs_log_recover_inode_skip(log, in_f);
+				error = 0;
+				goto out_release;
+			}
 		}
+
+		/* Take the opportunity to reset the flush iteration count */
+		ldip->di_flushiter = 0;
 	}
 
-	/* Take the opportunity to reset the flush iteration count */
-	ldip->di_flushiter = 0;
 
 	if (unlikely(S_ISREG(ldip->di_mode))) {
 		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
Signature · 1 line
2.42.0
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] xfs: recovery should not clear di_flushiter unconditionally

Quoted 55 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Because on v3 inodes, di_flushiter doesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.

This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.

This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing di_flushiter on v2 inodes during recovery.

Fixes 9b7d16e34bbe ("xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers")
---
 fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
fs/xfs/xfs_inode_item_recover.c +8 -13
index f4c31c2b60d5..dbdab4ce7c44 100644
@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(
 	 * superblock flag to determine whether we need to look at di_flushiter
 	 * to skip replay when the on disk inode is newer than the log one
 	 */
-	if (!xfs_has_v3inodes(mp) &&
-	    ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
-		/*
-		 * Deal with the wrap case, DI_MAX_FLUSH is less
-		 * than smaller numbers
-		 */
-		if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
-		    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
-			/* do nothing */
-		} else {
-			trace_xfs_log_recover_inode_skip(log, in_f);
-			error = 0;
-			goto out_release;
+	if (!xfs_has_v3inodes(mp)) {
+		if (ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
+			/*
+			 * Deal with the wrap case, DI_MAX_FLUSH is less
+			 * than smaller numbers
+			 */
+			if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
+			    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {

fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 391

Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +9 -0
+				/* do nothing */
+			} else {
+				trace_xfs_log_recover_inode_skip(log, in_f);
+				error = 0;
+				goto out_release;
+			}
 		}
+
+		/* Take the opportunity to reset the flush iteration count */
+		ldip->di_flushiter = 0;

Hmm. Well this fixes the zeroday problem, so thank you for getting the root of this!

Though hch did suggest reducing the amount of indenting here by compressing the if tests together. I can't decide if it's worth rearranging that old V4 code since none of it's scheduled for removal until 2030, but it /is/ legacy code that maybe we just don't care to touch?

<shrug>

--D

Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +0 -2
 	}
 
-	/* Take the opportunity to reset the flush iteration count */
-	ldip->di_flushiter = 0;
 
 	if (unlikely(S_ISREG(ldip->di_mode))) {
 		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
Signature · 1 line
2.42.0
#7 lore raw

Re: [PATCH 2/2] xfs: recovery should not clear di_flushiter unconditionally

Quoted 70 lines · Darrick J. Wong
Quoted 55 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Because on v3 inodes, di_flushiter doesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.

This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.

This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing di_flushiter on v2 inodes during recovery.

Fixes 9b7d16e34bbe ("xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers")
---
 fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
fs/xfs/xfs_inode_item_recover.c +8 -13
index f4c31c2b60d5..dbdab4ce7c44 100644
@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(
 	 * superblock flag to determine whether we need to look at di_flushiter
 	 * to skip replay when the on disk inode is newer than the log one
 	 */
-	if (!xfs_has_v3inodes(mp) &&
-	    ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
-		/*
-		 * Deal with the wrap case, DI_MAX_FLUSH is less
-		 * than smaller numbers
-		 */
-		if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
-		    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
-			/* do nothing */
-		} else {
-			trace_xfs_log_recover_inode_skip(log, in_f);
-			error = 0;
-			goto out_release;
+	if (!xfs_has_v3inodes(mp)) {
+		if (ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
+			/*
+			 * Deal with the wrap case, DI_MAX_FLUSH is less
+			 * than smaller numbers
+			 */
+			if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
+			    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {

fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 391

Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +9 -0
+				/* do nothing */
+			} else {
+				trace_xfs_log_recover_inode_skip(log, in_f);
+				error = 0;
+				goto out_release;
+			}
 		}
+
+		/* Take the opportunity to reset the flush iteration count */
+		ldip->di_flushiter = 0;

Hmm. Well this fixes the zeroday problem, so thank you for getting the root of this!

Quoted 7 lines

Though hch did suggest reducing the amount of indenting here by compressing the if tests together. I can't decide if it's worth rearranging that old V4 code since none of it's scheduled for removal until 2030, but it /is/ legacy code that maybe we just don't care to touch?

I'd prefer not to touch it. It's now all isolated in a "!xfs_has_v3inodes()" branch, so I think we can largely ignore the grot for now. If we have to do a larger rework of this code in future, then we can look at reworking it.

But right now, I really don't feel like risking breaking something else by doing unnecessary cleanups on code we haven't needed to touch in over a decade.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]