Skip to contentSkip to conversation

xfs: random patches on log recovery

PATCH New Gao Xiang 3 participants 11 messages Sep 17, 2020 – Sep 23, 2020 XFS filesystem

lore
Share & tools
lore

Gao Xiang

author
#0 lore raw

v2 cover xfs: random patches on log recovery

Hi folks,

Here are some patches after I read recovery code days ago. Due to code coupling, I send them as a patchset.

This version mainly addresses previous review from Brian, sorry for taking some time due to another work.

I already ran fstests and no strange out on my side and detail changelog is in each individual patch.

Thanks,
Gao Xiang

Gao Xiang (2):
xfs: avoid LR buffer overrun due to crafted h_len
xfs: clean up calculation of LR header blocks

 fs/xfs/xfs_log.c         |  4 +-
 fs/xfs/xfs_log_recover.c | 87 ++++++++++++++++------------------------
 2 files changed, 36 insertions(+), 55 deletions(-)
Signature · 1 line
2.18.1
author
#1 lore raw

PATCH v3 2/2 xfs: clean up calculation of LR header blocks

Let's use DIV_ROUND_UP() to calculate log record header blocks as what did in xlog_get_iclog_buffer_size() and wrap up a common helper for log recovery.

---
v2: https://lore.kernel.org/r/[email protected]

changes since v2:
- get rid of xlog_logrecv2_hblks() and use xlog_logrec_hblks() entirely (Brian).

 fs/xfs/xfs_log.c         |  4 +---
 fs/xfs/xfs_log_recover.c | 48 ++++++++++++++--------------------------
 2 files changed, 17 insertions(+), 35 deletions(-)
fs/xfs/xfs_log.c +1 -3
index ad0c69ee8947..7a4ba408a3a2 100644
@@ -1604,9 +1604,7 @@ xlog_cksum(
 		int		i;
 		int		xheads;
 
-		xheads = size / XLOG_HEADER_CYCLE_SIZE;
-		if (size % XLOG_HEADER_CYCLE_SIZE)
-			xheads++;
+		xheads = DIV_ROUND_UP(size, XLOG_HEADER_CYCLE_SIZE);
 
 		for (i = 1; i < xheads; i++) {
 			crc = crc32c(crc, &xhdr[i].hic_xheader,
fs/xfs/xfs_log_recover.c +16 -32

74 lines not on this page · raw message

Signature · 1 line
2.18.1
Reviewed-by on v3 2/2
#2 lore raw

Re: [PATCH v3 2/2] xfs: clean up calculation of LR header blocks

Quoted 36 lines · Gao Xiang

Let's use DIV_ROUND_UP() to calculate log record header blocks as what did in xlog_get_iclog_buffer_size() and wrap up a common helper for log recovery.

---
v2: https://lore.kernel.org/r/[email protected]

changes since v2:
- get rid of xlog_logrecv2_hblks() and use xlog_logrec_hblks() entirely (Brian).

 fs/xfs/xfs_log.c         |  4 +---
 fs/xfs/xfs_log_recover.c | 48 ++++++++++++++--------------------------
 2 files changed, 17 insertions(+), 35 deletions(-)
fs/xfs/xfs_log.c +1 -3
index ad0c69ee8947..7a4ba408a3a2 100644
@@ -1604,9 +1604,7 @@ xlog_cksum(
 		int		i;
 		int		xheads;
 
-		xheads = size / XLOG_HEADER_CYCLE_SIZE;
-		if (size % XLOG_HEADER_CYCLE_SIZE)
-			xheads++;
+		xheads = DIV_ROUND_UP(size, XLOG_HEADER_CYCLE_SIZE);
 
 		for (i = 1; i < xheads; i++) {
 			crc = crc32c(crc, &xhdr[i].hic_xheader,
fs/xfs/xfs_log_recover.c +0 -0
index 782ec3eeab4d..28dd98b5a703 100644

fs/xfs/xfs_log_recover.c xlog_find_verify_cycle( · line 375

Quoted 5 lines
fs/xfs/xfs_log_recover.c +2 -0
@@ -371,6 +371,18 @@ xlog_find_verify_cycle(
 	return error;
 }
 
+static inline int xlog_logrec_hblks(struct xlog *log, xlog_rec_header_t *rh)
+{

We're trying to gradually eliminate various structure typedefs so it's frowned upon to introduce new instances. If you replace the above with 'struct xlog_rec_header,' the rest looks good to me:

Quoted 75 lines
fs/xfs/xfs_log_recover.c +14 -32
+	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
+		int	h_size = be32_to_cpu(rh->h_size);
+
+		if ((be32_to_cpu(rh->h_version) & XLOG_VERSION_2) &&
+		    h_size > XLOG_HEADER_CYCLE_SIZE)
+			return DIV_ROUND_UP(h_size, XLOG_HEADER_CYCLE_SIZE);
+	}
+	return 1;
+}
+
 /*
  * Potentially backup over partial log record write.
  *
@@ -463,15 +475,7 @@ xlog_find_verify_log_record(
 	 * reset last_blk.  Only when last_blk points in the middle of a log
 	 * record do we update last_blk.
 	 */
-	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
-		uint	h_size = be32_to_cpu(head->h_size);
-
-		xhdrs = h_size / XLOG_HEADER_CYCLE_SIZE;
-		if (h_size % XLOG_HEADER_CYCLE_SIZE)
-			xhdrs++;
-	} else {
-		xhdrs = 1;
-	}
+	xhdrs = xlog_logrec_hblks(log, head);
 
 	if (*last_blk - i + extra_bblks !=
 	    BTOBB(be32_to_cpu(head->h_len)) + xhdrs)
@@ -1158,22 +1162,7 @@ xlog_check_unmount_rec(
 	 * below. We won't want to clear the unmount record if there is one, so
 	 * we pass the lsn of the unmount record rather than the block after it.
 	 */
-	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
-		int	h_size = be32_to_cpu(rhead->h_size);
-		int	h_version = be32_to_cpu(rhead->h_version);
-
-		if ((h_version & XLOG_VERSION_2) &&
-		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
-			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
-			if (h_size % XLOG_HEADER_CYCLE_SIZE)
-				hblks++;
-		} else {
-			hblks = 1;
-		}
-	} else {
-		hblks = 1;
-	}
-
+	hblks = xlog_logrec_hblks(log, rhead);
 	after_umount_blk = xlog_wrap_logbno(log,
 			rhead_blk + hblks + BTOBB(be32_to_cpu(rhead->h_len)));
 
@@ -2989,15 +2978,10 @@ xlog_do_recovery_pass(
 		if (error)
 			goto bread_err1;
 
-		if ((be32_to_cpu(rhead->h_version) & XLOG_VERSION_2) &&
-		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
-			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
-			if (h_size % XLOG_HEADER_CYCLE_SIZE)
-				hblks++;
+		hblks = xlog_logrec_hblks(log, rhead);
+		if (hblks != 1) {
 			kmem_free(hbp);
 			hbp = xlog_alloc_buffer(log, hblks);
-		} else {
-			hblks = 1;
 		}
 	} else {
 		ASSERT(log->l_sectBBsize == 1);
Signature · 1 line
2.18.1
author
#3 lore raw

Re: [PATCH v3 2/2] xfs: clean up calculation of LR header blocks

Hi Brian,

Quoted 40 lines · Brian Foster
Quoted 39 lines · Gao Xiang

Let's use DIV_ROUND_UP() to calculate log record header blocks as what did in xlog_get_iclog_buffer_size() and wrap up a common helper for log recovery.

---
v2: https://lore.kernel.org/r/[email protected]

changes since v2:
- get rid of xlog_logrecv2_hblks() and use xlog_logrec_hblks() entirely (Brian).

 fs/xfs/xfs_log.c         |  4 +---
 fs/xfs/xfs_log_recover.c | 48 ++++++++++++++--------------------------
 2 files changed, 17 insertions(+), 35 deletions(-)
fs/xfs/xfs_log.c +1 -3
index ad0c69ee8947..7a4ba408a3a2 100644
@@ -1604,9 +1604,7 @@ xlog_cksum(
 		int		i;
 		int		xheads;
 
-		xheads = size / XLOG_HEADER_CYCLE_SIZE;
-		if (size % XLOG_HEADER_CYCLE_SIZE)
-			xheads++;
+		xheads = DIV_ROUND_UP(size, XLOG_HEADER_CYCLE_SIZE);
 
 		for (i = 1; i < xheads; i++) {
 			crc = crc32c(crc, &xhdr[i].hic_xheader,
fs/xfs/xfs_log_recover.c +0 -0
index 782ec3eeab4d..28dd98b5a703 100644
@@ -371,6 +371,18 @@ xlog_find_verify_cycle(
 	return error;
 }
 
Quoted 8 lines

fs/xfs/xfs_log_recover.c xlog_find_verify_cycle( · line 375

Quoted 2 lines
fs/xfs/xfs_log_recover.c +2 -0
+static inline int xlog_logrec_hblks(struct xlog *log, xlog_rec_header_t *rh)
+{

We're trying to gradually eliminate various structure typedefs so it's frowned upon to introduce new instances. If you replace the above with 'struct xlog_rec_header,' the rest looks good to me:

Thanks for your review. Yeah, I noticed the eliminate movement and I used xlog_rec_header_t by mistake.

I will resend a follow-up "in-reply-to" patch to fix that.

Thanks,
Gao Xiang

author
#4 lore raw

PATCH 1/1 xfs: clean up calculation of LR header blocks

In reply to Gao Xiang

Let's use DIV_ROUND_UP() to calculate log record header blocks as what did in xlog_get_iclog_buffer_size() and wrap up a common helper for log recovery.

---
changelog:
- 'xlog_rec_header_t' => 'struct xlog_rec_header' to eliminate various structure typedefs (Brian).

 fs/xfs/xfs_log.c         |  4 +---
 fs/xfs/xfs_log_recover.c | 49 ++++++++++++++--------------------------
 2 files changed, 18 insertions(+), 35 deletions(-)
fs/xfs/xfs_log.c +1 -3
index ad0c69ee8947..7a4ba408a3a2 100644
@@ -1604,9 +1604,7 @@ xlog_cksum(
 		int		i;
 		int		xheads;
 
-		xheads = size / XLOG_HEADER_CYCLE_SIZE;
-		if (size % XLOG_HEADER_CYCLE_SIZE)
-			xheads++;
+		xheads = DIV_ROUND_UP(size, XLOG_HEADER_CYCLE_SIZE);
 
 		for (i = 1; i < xheads; i++) {
 			crc = crc32c(crc, &xhdr[i].hic_xheader,
fs/xfs/xfs_log_recover.c +17 -32

75 lines not on this page · raw message

Signature · 1 line
2.18.1
Reviewed-by on v1 1/1
#5 lore raw

Re: [PATCH v3.2] xfs: clean up calculation of LR header blocks

Quoted 6 lines · Gao Xiang

Let's use DIV_ROUND_UP() to calculate log record header blocks as what did in xlog_get_iclog_buffer_size() and wrap up a common helper for log recovery.

Looks ok,

--D

Quoted 110 lines

---
changelog:
- 'xlog_rec_header_t' => 'struct xlog_rec_header' to eliminate various structure typedefs (Brian).

 fs/xfs/xfs_log.c         |  4 +---
 fs/xfs/xfs_log_recover.c | 49 ++++++++++++++--------------------------
 2 files changed, 18 insertions(+), 35 deletions(-)
fs/xfs/xfs_log.c +1 -3
index ad0c69ee8947..7a4ba408a3a2 100644
@@ -1604,9 +1604,7 @@ xlog_cksum(
 		int		i;
 		int		xheads;
 
-		xheads = size / XLOG_HEADER_CYCLE_SIZE;
-		if (size % XLOG_HEADER_CYCLE_SIZE)
-			xheads++;
+		xheads = DIV_ROUND_UP(size, XLOG_HEADER_CYCLE_SIZE);
 
 		for (i = 1; i < xheads; i++) {
 			crc = crc32c(crc, &xhdr[i].hic_xheader,
fs/xfs/xfs_log_recover.c +17 -32
index 782ec3eeab4d..12cde89c090b 100644
@@ -371,6 +371,19 @@ xlog_find_verify_cycle(
 	return error;
 }
 
+static inline int
+xlog_logrec_hblks(struct xlog *log, struct xlog_rec_header *rh)
+{
+	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
+		int	h_size = be32_to_cpu(rh->h_size);
+
+		if ((be32_to_cpu(rh->h_version) & XLOG_VERSION_2) &&
+		    h_size > XLOG_HEADER_CYCLE_SIZE)
+			return DIV_ROUND_UP(h_size, XLOG_HEADER_CYCLE_SIZE);
+	}
+	return 1;
+}
+
 /*
  * Potentially backup over partial log record write.
  *
@@ -463,15 +476,7 @@ xlog_find_verify_log_record(
 	 * reset last_blk.  Only when last_blk points in the middle of a log
 	 * record do we update last_blk.
 	 */
-	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
-		uint	h_size = be32_to_cpu(head->h_size);
-
-		xhdrs = h_size / XLOG_HEADER_CYCLE_SIZE;
-		if (h_size % XLOG_HEADER_CYCLE_SIZE)
-			xhdrs++;
-	} else {
-		xhdrs = 1;
-	}
+	xhdrs = xlog_logrec_hblks(log, head);
 
 	if (*last_blk - i + extra_bblks !=
 	    BTOBB(be32_to_cpu(head->h_len)) + xhdrs)
@@ -1158,22 +1163,7 @@ xlog_check_unmount_rec(
 	 * below. We won't want to clear the unmount record if there is one, so
 	 * we pass the lsn of the unmount record rather than the block after it.
 	 */
-	if (xfs_sb_version_haslogv2(&log->l_mp->m_sb)) {
-		int	h_size = be32_to_cpu(rhead->h_size);
-		int	h_version = be32_to_cpu(rhead->h_version);
-
-		if ((h_version & XLOG_VERSION_2) &&
-		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
-			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
-			if (h_size % XLOG_HEADER_CYCLE_SIZE)
-				hblks++;
-		} else {
-			hblks = 1;
-		}
-	} else {
-		hblks = 1;
-	}
-
+	hblks = xlog_logrec_hblks(log, rhead);
 	after_umount_blk = xlog_wrap_logbno(log,
 			rhead_blk + hblks + BTOBB(be32_to_cpu(rhead->h_len)));
 
@@ -2989,15 +2979,10 @@ xlog_do_recovery_pass(
 		if (error)
 			goto bread_err1;
 
-		if ((be32_to_cpu(rhead->h_version) & XLOG_VERSION_2) &&
-		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
-			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
-			if (h_size % XLOG_HEADER_CYCLE_SIZE)
-				hblks++;
+		hblks = xlog_logrec_hblks(log, rhead);
+		if (hblks != 1) {
 			kmem_free(hbp);
 			hbp = xlog_alloc_buffer(log, hblks);
-		} else {
-			hblks = 1;
 		}
 	} else {
 		ASSERT(log->l_sectBBsize == 1);
Signature · 1 line
2.18.1
author
#6 lore raw

PATCH v4 1/2 xfs: avoid LR buffer overrun due to crafted h_len

In reply to Gao Xiang

Currently, crafted h_len has been blocked for the log header of the tail block in commit a70f9fe52daa ("xfs:
detect and handle invalid iclog size set by mkfs").

However, each log record could still have crafted h_len and cause log record buffer overrun. So let's check h_len vs buffer size for each log record as well.

---
v3: https://lore.kernel.org/r/[email protected]

changes since v3:
- drop exception comment to avoid confusion (Brian);
- check rhead->hlen vs buffer size to address the harmful overflow (Brian);

And as Brian requested previously, "Also, please test the workaround case to make sure it still works as expected (if you haven't already)."

So I tested the vanilla/after upstream kernels with compiled xfsprogs-v4.3.0, which was before mkfs fix 20fbd4593ff2 ("libxfs: format the log with valid log record headers") got merged, and I generated a questionable image followed by the instruction described in the related commit messages with "mkfs.xfs -dsunit=512,swidth=4096 -f test.img" and logprint says

cycle: 1 version: 2 lsn: 1,0 tail_lsn: 1,0
length of Log Record: 261632 prev offset: -1 num ops: 1
uuid: 7b84cd80-7855-4dc8-91ce-542c7d65ba99 format: little endian linux
h_size: 32768

so "length of Log Record" is overrun obviously, but I cannot reproduce the described workaround case for vanilla/after kernels anymore.

I think the reason is due to commit 7f6aff3a29b0 ("xfs: only run torn log write detection on dirty logs"), which changes the behavior described in commit a70f9fe52daa8 ("xfs: detect and handle invalid iclog size set by mkfs") from "all records at the head of the log are verified for CRC errors" to "we can only run CRC verification when the log is dirty because there's no guarantee that the log data behind an unmount record is compatible with the current architecture).", more details see codediff of a70f9fe52daa8.

The timeline seems to be:
https://lore.kernel.org/r/[email protected]
a70f9fe52daa [PATCH v2 1/8] xfs: detect and handle invalid iclog size set by mkfs 7088c4136fa1 [PATCH v2 7/8] xfs: detect and trim torn writes during log recovery
https://lore.kernel.org/r/[email protected]
7f6aff3a29b0 [PATCH 4/4] xfs: only run torn log write detection on dirty logs

so IMHO, the workaround a70f9fe52daa would only be useful between 7088c4136fa1 ~ 7f6aff3a29b0.

Yeah, it relates to several old kernel commits/versions, my technical analysis is as above. This patch doesn't actually change the real original workaround logic. Even if the workground can be removed now, that should be addressed with another patch and that is quite another story.

Kindly correct me if I'm wrong :-)

Thanks,
Gao Xiang

 fs/xfs/xfs_log_recover.c | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)
fs/xfs/xfs_log_recover.c +19 -20
index a17d788921d6..782ec3eeab4d 100644
@@ -2878,7 +2878,8 @@ STATIC int
 xlog_valid_rec_header(
 	struct xlog		*log,
 	struct xlog_rec_header	*rhead,
-	xfs_daddr_t		blkno)
+	xfs_daddr_t		blkno,
+	int			bufsize)
 {
 	int			hlen;
 
@@ -2894,10 +2895,14 @@ xlog_valid_rec_header(
 		return -EFSCORRUPTED;
 	}
 
-	/* LR body must have data or it wouldn't have been written */
+	/*
+	 * LR body must have data (or it wouldn't have been written)
+	 * and h_len must not be greater than LR buffer size.
+	 */
 	hlen = be32_to_cpu(rhead->h_len);
-	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > INT_MAX))
+	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > bufsize))
 		return -EFSCORRUPTED;
+
 	if (XFS_IS_CORRUPT(log->l_mp,
 			   blkno > log->l_logBBsize || blkno > INT_MAX))
 		return -EFSCORRUPTED;
@@ -2958,9 +2963,6 @@ xlog_do_recovery_pass(
 			goto bread_err1;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, tail_blk);
-		if (error)
-			goto bread_err1;
 
 		/*
 		 * xfsprogs has a bug where record length is based on lsunit but
@@ -2975,21 +2977,18 @@ xlog_do_recovery_pass(
 		 */
 		h_size = be32_to_cpu(rhead->h_size);
 		h_len = be32_to_cpu(rhead->h_len);
-		if (h_len > h_size) {
-			if (h_len <= log->l_mp->m_logbsize &&
-			    be32_to_cpu(rhead->h_num_logops) == 1) {
-				xfs_warn(log->l_mp,
+		if (h_len > h_size && h_len <= log->l_mp->m_logbsize &&
+		    rhead->h_num_logops == cpu_to_be32(1)) {
+			xfs_warn(log->l_mp,
 		"invalid iclog size (%d bytes), using lsunit (%d bytes)",
-					 h_size, log->l_mp->m_logbsize);
-				h_size = log->l_mp->m_logbsize;
-			} else {
-				XFS_ERROR_REPORT(__func__, XFS_ERRLEVEL_LOW,
-						log->l_mp);
-				error = -EFSCORRUPTED;
-				goto bread_err1;
-			}
+				 h_size, log->l_mp->m_logbsize);
+			h_size = log->l_mp->m_logbsize;
 		}
 
+		error = xlog_valid_rec_header(log, rhead, tail_blk, h_size);
+		if (error)
+			goto bread_err1;
+
 		if ((be32_to_cpu(rhead->h_version) & XLOG_VERSION_2) &&
 		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
 			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
@@ -3070,7 +3069,7 @@ xlog_do_recovery_pass(
 			}
 			rhead = (xlog_rec_header_t *)offset;
 			error = xlog_valid_rec_header(log, rhead,
-						split_hblks ? blk_no : 0);
+					split_hblks ? blk_no : 0, h_size);
 			if (error)
 				goto bread_err2;
 
@@ -3151,7 +3150,7 @@ xlog_do_recovery_pass(
 			goto bread_err2;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, blk_no);
+		error = xlog_valid_rec_header(log, rhead, blk_no, h_size);
 		if (error)
 			goto bread_err2;
 
Signature · 1 line
2.18.1
Reviewed-by on v4 1/2
#7 lore raw

Re: [PATCH v4 1/2] xfs: avoid LR buffer overrun due to crafted h_len

Quoted 35 lines · Gao Xiang

Currently, crafted h_len has been blocked for the log header of the tail block in commit a70f9fe52daa ("xfs:
detect and handle invalid iclog size set by mkfs").

However, each log record could still have crafted h_len and cause log record buffer overrun. So let's check h_len vs buffer size for each log record as well.

---
v3: https://lore.kernel.org/r/[email protected]

changes since v3:
- drop exception comment to avoid confusion (Brian);
- check rhead->hlen vs buffer size to address the harmful overflow (Brian);

And as Brian requested previously, "Also, please test the workaround case to make sure it still works as expected (if you haven't already)."

So I tested the vanilla/after upstream kernels with compiled xfsprogs-v4.3.0, which was before mkfs fix 20fbd4593ff2 ("libxfs: format the log with valid log record headers") got merged, and I generated a questionable image followed by the instruction described in the related commit messages with "mkfs.xfs -dsunit=512,swidth=4096 -f test.img" and logprint says

cycle: 1 version: 2 lsn: 1,0 tail_lsn: 1,0
length of Log Record: 261632 prev offset: -1 num ops: 1
uuid: 7b84cd80-7855-4dc8-91ce-542c7d65ba99 format: little endian linux
h_size: 32768

so "length of Log Record" is overrun obviously, but I cannot reproduce the described workaround case for vanilla/after kernels anymore.

Quoted 9 lines

I think the reason is due to commit 7f6aff3a29b0 ("xfs: only run torn log write detection on dirty logs"), which changes the behavior described in commit a70f9fe52daa8 ("xfs: detect and handle invalid iclog size set by mkfs") from "all records at the head of the log are verified for CRC errors" to "we can only run CRC verification when the log is dirty because there's no guarantee that the log data behind an unmount record is compatible with the current architecture).", more details see codediff of a70f9fe52daa8.

If I follow correctly, you're saying that prior to commit 7f6aff3a29b0, log recovery would run a CRC pass on a clean log (with an unmount record) and this is where the old workaround code would kick in if the filesystem happened to be misformatted by mkfs. After said commit, the CRC pass is no longer run unless the log is dirty (for arch compatibility reasons), so we fall into the xlog_check_unmount_rec() path that does some careful (presumably arch agnostic) detection of a clean/dirty log based on whether the record just behind the head has a single unmount transaction. This function already uses h_len properly and only reads a single log block to determine whether the target is an unmount record, so doesn't have the same overflow risk as a full recovery pass.

Am I following that correctly? If so, the patch otherwise looks reasonable to me:

Quoted 117 lines

The timeline seems to be:
https://lore.kernel.org/r/[email protected]
a70f9fe52daa [PATCH v2 1/8] xfs: detect and handle invalid iclog size set by mkfs 7088c4136fa1 [PATCH v2 7/8] xfs: detect and trim torn writes during log recovery
https://lore.kernel.org/r/[email protected]
7f6aff3a29b0 [PATCH 4/4] xfs: only run torn log write detection on dirty logs

so IMHO, the workaround a70f9fe52daa would only be useful between 7088c4136fa1 ~ 7f6aff3a29b0.

Yeah, it relates to several old kernel commits/versions, my technical analysis is as above. This patch doesn't actually change the real original workaround logic. Even if the workground can be removed now, that should be addressed with another patch and that is quite another story.

Kindly correct me if I'm wrong :-)

Thanks,
Gao Xiang

 fs/xfs/xfs_log_recover.c | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)
fs/xfs/xfs_log_recover.c +19 -20
index a17d788921d6..782ec3eeab4d 100644
@@ -2878,7 +2878,8 @@ STATIC int
 xlog_valid_rec_header(
 	struct xlog		*log,
 	struct xlog_rec_header	*rhead,
-	xfs_daddr_t		blkno)
+	xfs_daddr_t		blkno,
+	int			bufsize)
 {
 	int			hlen;
 
@@ -2894,10 +2895,14 @@ xlog_valid_rec_header(
 		return -EFSCORRUPTED;
 	}
 
-	/* LR body must have data or it wouldn't have been written */
+	/*
+	 * LR body must have data (or it wouldn't have been written)
+	 * and h_len must not be greater than LR buffer size.
+	 */
 	hlen = be32_to_cpu(rhead->h_len);
-	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > INT_MAX))
+	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > bufsize))
 		return -EFSCORRUPTED;
+
 	if (XFS_IS_CORRUPT(log->l_mp,
 			   blkno > log->l_logBBsize || blkno > INT_MAX))
 		return -EFSCORRUPTED;
@@ -2958,9 +2963,6 @@ xlog_do_recovery_pass(
 			goto bread_err1;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, tail_blk);
-		if (error)
-			goto bread_err1;
 
 		/*
 		 * xfsprogs has a bug where record length is based on lsunit but
@@ -2975,21 +2977,18 @@ xlog_do_recovery_pass(
 		 */
 		h_size = be32_to_cpu(rhead->h_size);
 		h_len = be32_to_cpu(rhead->h_len);
-		if (h_len > h_size) {
-			if (h_len <= log->l_mp->m_logbsize &&
-			    be32_to_cpu(rhead->h_num_logops) == 1) {
-				xfs_warn(log->l_mp,
+		if (h_len > h_size && h_len <= log->l_mp->m_logbsize &&
+		    rhead->h_num_logops == cpu_to_be32(1)) {
+			xfs_warn(log->l_mp,
 		"invalid iclog size (%d bytes), using lsunit (%d bytes)",
-					 h_size, log->l_mp->m_logbsize);
-				h_size = log->l_mp->m_logbsize;
-			} else {
-				XFS_ERROR_REPORT(__func__, XFS_ERRLEVEL_LOW,
-						log->l_mp);
-				error = -EFSCORRUPTED;
-				goto bread_err1;
-			}
+				 h_size, log->l_mp->m_logbsize);
+			h_size = log->l_mp->m_logbsize;
 		}
 
+		error = xlog_valid_rec_header(log, rhead, tail_blk, h_size);
+		if (error)
+			goto bread_err1;
+
 		if ((be32_to_cpu(rhead->h_version) & XLOG_VERSION_2) &&
 		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
 			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
@@ -3070,7 +3069,7 @@ xlog_do_recovery_pass(
 			}
 			rhead = (xlog_rec_header_t *)offset;
 			error = xlog_valid_rec_header(log, rhead,
-						split_hblks ? blk_no : 0);
+					split_hblks ? blk_no : 0, h_size);
 			if (error)
 				goto bread_err2;
 
@@ -3151,7 +3150,7 @@ xlog_do_recovery_pass(
 			goto bread_err2;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, blk_no);
+		error = xlog_valid_rec_header(log, rhead, blk_no, h_size);
 		if (error)
 			goto bread_err2;
 
Signature · 1 line
2.18.1
author
#8 lore raw

Re: [PATCH v4 1/2] xfs: avoid LR buffer overrun due to crafted h_len

Hi Brian,

Quoted 59 lines · Brian Foster
Quoted 35 lines · Gao Xiang

Currently, crafted h_len has been blocked for the log header of the tail block in commit a70f9fe52daa ("xfs:
detect and handle invalid iclog size set by mkfs").

However, each log record could still have crafted h_len and cause log record buffer overrun. So let's check h_len vs buffer size for each log record as well.

---
v3: https://lore.kernel.org/r/[email protected]

changes since v3:
- drop exception comment to avoid confusion (Brian);
- check rhead->hlen vs buffer size to address the harmful overflow (Brian);

And as Brian requested previously, "Also, please test the workaround case to make sure it still works as expected (if you haven't already)."

So I tested the vanilla/after upstream kernels with compiled xfsprogs-v4.3.0, which was before mkfs fix 20fbd4593ff2 ("libxfs: format the log with valid log record headers") got merged, and I generated a questionable image followed by the instruction described in the related commit messages with "mkfs.xfs -dsunit=512,swidth=4096 -f test.img" and logprint says

cycle: 1 version: 2 lsn: 1,0 tail_lsn: 1,0
length of Log Record: 261632 prev offset: -1 num ops: 1
uuid: 7b84cd80-7855-4dc8-91ce-542c7d65ba99 format: little endian linux
h_size: 32768

so "length of Log Record" is overrun obviously, but I cannot reproduce the described workaround case for vanilla/after kernels anymore.

Quoted 9 lines

I think the reason is due to commit 7f6aff3a29b0 ("xfs: only run torn log write detection on dirty logs"), which changes the behavior described in commit a70f9fe52daa8 ("xfs: detect and handle invalid iclog size set by mkfs") from "all records at the head of the log are verified for CRC errors" to "we can only run CRC verification when the log is dirty because there's no guarantee that the log data behind an unmount record is compatible with the current architecture).", more details see codediff of a70f9fe52daa8.

If I follow correctly, you're saying that prior to commit 7f6aff3a29b0, log recovery would run a CRC pass on a clean log (with an unmount record) and this is where the old workaround code would kick in if the filesystem happened to be misformatted by mkfs. After said commit, the CRC pass is no longer run unless the log is dirty (for arch compatibility reasons), so we fall into the xlog_check_unmount_rec() path that does some careful (presumably arch agnostic) detection of a clean/dirty log based on whether the record just behind the head has a single unmount transaction. This function already uses h_len properly and only reads a single log block to determine whether the target is an unmount record, so doesn't have the same overflow risk as a full recovery pass.

Quoted 2 lines

Am I following that correctly? If so, the patch otherwise looks reasonable to me:

Yeah, that is what I was trying to say. Thanks for the review!

Thanks,
Gao Xiang

Quoted 23 lines
Quoted 20 lines

The timeline seems to be:
https://lore.kernel.org/r/[email protected]
a70f9fe52daa [PATCH v2 1/8] xfs: detect and handle invalid iclog size set by mkfs 7088c4136fa1 [PATCH v2 7/8] xfs: detect and trim torn writes during log recovery
https://lore.kernel.org/r/[email protected]
7f6aff3a29b0 [PATCH 4/4] xfs: only run torn log write detection on dirty logs

so IMHO, the workaround a70f9fe52daa would only be useful between 7088c4136fa1 ~ 7f6aff3a29b0.

Yeah, it relates to several old kernel commits/versions, my technical analysis is as above. This patch doesn't actually change the real original workaround logic. Even if the workground can be removed now, that should be addressed with another patch and that is quite another story.

Kindly correct me if I'm wrong :-)

Thanks,
Gao Xiang

Reviewed-by on v4 1/2
#9 lore raw

Re: [PATCH v4 1/2] xfs: avoid LR buffer overrun due to crafted h_len

In reply to Gao Xiang
Quoted 9 lines · Gao Xiang

Currently, crafted h_len has been blocked for the log header of the tail block in commit a70f9fe52daa ("xfs:
detect and handle invalid iclog size set by mkfs").

However, each log record could still have crafted h_len and cause log record buffer overrun. So let's check h_len vs buffer size for each log record as well.

/me squints real hard and thinks he understands what this patch does.

Tighten up xlog_valid_rec_header, and add a new callsite in the middle of xlog_do_recovery_pass instead of the insufficient length checking. Assuming that's right,

--D

Quoted 152 lines

---
v3: https://lore.kernel.org/r/[email protected]

changes since v3:
- drop exception comment to avoid confusion (Brian);
- check rhead->hlen vs buffer size to address the harmful overflow (Brian);

And as Brian requested previously, "Also, please test the workaround case to make sure it still works as expected (if you haven't already)."

So I tested the vanilla/after upstream kernels with compiled xfsprogs-v4.3.0, which was before mkfs fix 20fbd4593ff2 ("libxfs: format the log with valid log record headers") got merged, and I generated a questionable image followed by the instruction described in the related commit messages with "mkfs.xfs -dsunit=512,swidth=4096 -f test.img" and logprint says

cycle: 1 version: 2 lsn: 1,0 tail_lsn: 1,0
length of Log Record: 261632 prev offset: -1 num ops: 1
uuid: 7b84cd80-7855-4dc8-91ce-542c7d65ba99 format: little endian linux
h_size: 32768

so "length of Log Record" is overrun obviously, but I cannot reproduce the described workaround case for vanilla/after kernels anymore.

I think the reason is due to commit 7f6aff3a29b0 ("xfs: only run torn log write detection on dirty logs"), which changes the behavior described in commit a70f9fe52daa8 ("xfs: detect and handle invalid iclog size set by mkfs") from "all records at the head of the log are verified for CRC errors" to "we can only run CRC verification when the log is dirty because there's no guarantee that the log data behind an unmount record is compatible with the current architecture).", more details see codediff of a70f9fe52daa8.

The timeline seems to be:
https://lore.kernel.org/r/[email protected]
a70f9fe52daa [PATCH v2 1/8] xfs: detect and handle invalid iclog size set by mkfs 7088c4136fa1 [PATCH v2 7/8] xfs: detect and trim torn writes during log recovery
https://lore.kernel.org/r/[email protected]
7f6aff3a29b0 [PATCH 4/4] xfs: only run torn log write detection on dirty logs

so IMHO, the workaround a70f9fe52daa would only be useful between 7088c4136fa1 ~ 7f6aff3a29b0.

Yeah, it relates to several old kernel commits/versions, my technical analysis is as above. This patch doesn't actually change the real original workaround logic. Even if the workground can be removed now, that should be addressed with another patch and that is quite another story.

Kindly correct me if I'm wrong :-)

Thanks,
Gao Xiang

 fs/xfs/xfs_log_recover.c | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)
fs/xfs/xfs_log_recover.c +19 -20
index a17d788921d6..782ec3eeab4d 100644
@@ -2878,7 +2878,8 @@ STATIC int
 xlog_valid_rec_header(
 	struct xlog		*log,
 	struct xlog_rec_header	*rhead,
-	xfs_daddr_t		blkno)
+	xfs_daddr_t		blkno,
+	int			bufsize)
 {
 	int			hlen;
 
@@ -2894,10 +2895,14 @@ xlog_valid_rec_header(
 		return -EFSCORRUPTED;
 	}
 
-	/* LR body must have data or it wouldn't have been written */
+	/*
+	 * LR body must have data (or it wouldn't have been written)
+	 * and h_len must not be greater than LR buffer size.
+	 */
 	hlen = be32_to_cpu(rhead->h_len);
-	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > INT_MAX))
+	if (XFS_IS_CORRUPT(log->l_mp, hlen <= 0 || hlen > bufsize))
 		return -EFSCORRUPTED;
+
 	if (XFS_IS_CORRUPT(log->l_mp,
 			   blkno > log->l_logBBsize || blkno > INT_MAX))
 		return -EFSCORRUPTED;
@@ -2958,9 +2963,6 @@ xlog_do_recovery_pass(
 			goto bread_err1;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, tail_blk);
-		if (error)
-			goto bread_err1;
 
 		/*
 		 * xfsprogs has a bug where record length is based on lsunit but
@@ -2975,21 +2977,18 @@ xlog_do_recovery_pass(
 		 */
 		h_size = be32_to_cpu(rhead->h_size);
 		h_len = be32_to_cpu(rhead->h_len);
-		if (h_len > h_size) {
-			if (h_len <= log->l_mp->m_logbsize &&
-			    be32_to_cpu(rhead->h_num_logops) == 1) {
-				xfs_warn(log->l_mp,
+		if (h_len > h_size && h_len <= log->l_mp->m_logbsize &&
+		    rhead->h_num_logops == cpu_to_be32(1)) {
+			xfs_warn(log->l_mp,
 		"invalid iclog size (%d bytes), using lsunit (%d bytes)",
-					 h_size, log->l_mp->m_logbsize);
-				h_size = log->l_mp->m_logbsize;
-			} else {
-				XFS_ERROR_REPORT(__func__, XFS_ERRLEVEL_LOW,
-						log->l_mp);
-				error = -EFSCORRUPTED;
-				goto bread_err1;
-			}
+				 h_size, log->l_mp->m_logbsize);
+			h_size = log->l_mp->m_logbsize;
 		}
 
+		error = xlog_valid_rec_header(log, rhead, tail_blk, h_size);
+		if (error)
+			goto bread_err1;
+
 		if ((be32_to_cpu(rhead->h_version) & XLOG_VERSION_2) &&
 		    (h_size > XLOG_HEADER_CYCLE_SIZE)) {
 			hblks = h_size / XLOG_HEADER_CYCLE_SIZE;
@@ -3070,7 +3069,7 @@ xlog_do_recovery_pass(
 			}
 			rhead = (xlog_rec_header_t *)offset;
 			error = xlog_valid_rec_header(log, rhead,
-						split_hblks ? blk_no : 0);
+					split_hblks ? blk_no : 0, h_size);
 			if (error)
 				goto bread_err2;
 
@@ -3151,7 +3150,7 @@ xlog_do_recovery_pass(
 			goto bread_err2;
 
 		rhead = (xlog_rec_header_t *)offset;
-		error = xlog_valid_rec_header(log, rhead, blk_no);
+		error = xlog_valid_rec_header(log, rhead, blk_no, h_size);
 		if (error)
 			goto bread_err2;
 
Signature · 1 line
2.18.1
author
#10 lore raw

Re: [PATCH v4 1/2] xfs: avoid LR buffer overrun due to crafted h_len

Quoted 11 lines · Darrick J. Wong
Quoted 9 lines · Gao Xiang

Currently, crafted h_len has been blocked for the log header of the tail block in commit a70f9fe52daa ("xfs:
detect and handle invalid iclog size set by mkfs").

However, each log record could still have crafted h_len and cause log record buffer overrun. So let's check h_len vs buffer size for each log record as well.

Quoted 7 lines

/me squints real hard and thinks he understands what this patch does.

Tighten up xlog_valid_rec_header, and add a new callsite in the middle of xlog_do_recovery_pass instead of the insufficient length checking. Assuming that's right,

Thanks for the review!

The main point is to check each individual LR h_len against LR buffer size allocated first in xlog_do_recovery_pass() to avoid buffer overflow triggered by CRC calc or follow-up steps (details in
https://lore.kernel.org/linux-xfs/[email protected]/ ).

no new callsite (just move the callsite after original workaround code). Anyway, that is just a buffer overflow issue can be triggered by corrupted log. Just a minor stuff but security guyes could also keep eyes on such thing. I think that is right. :)

Thanks,
Gao Xiang

Quoted 2 lines

--D