Skip to contentSkip to conversation

xfs_repair: fix xfs/033 issues.

PATCH In review Dave Chinner 3 participants 8 messages Feb 20, 2014

lore
Share & tools
lore
#0 lore raw

cover xfs_repair: fix xfs/033 issues.

Hi folks,

Eric reported that there was a regression in v3.2.0-alpha2 where xfs/033 was failing to repair the corrupted root inode. He also discovered that the discontiguous buffer support in xfs_repair fixed it. Turns out that was a regression in xfs_repair covering up another regression in xfs_repair.

Two bugs don't make broken code better.

The first patch fixes the regression that discontiguous buffer support introduced, re-exposing the original regression so it can be debugged. The second patch fixes the original regression.

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/2 libxfs: contiguous buffers are not discontigous

From: Dave Chinner <[email protected]>

When discontiguous directory buffer support was fixed in xfs_repair, (dd9093d xfs_repair: fix discontiguous directory block support) it changed to using libxfs_getbuf_map() to support mapping discontiguous blocks, and the prefetch code special cased such discontiguous buffers.

The issue is that libxfs_getbuf_map() marks all buffers, even contiguous ones - as LIBXFS_B_DISCONTIG, and so the prefetch code was treating every buffer as discontiguous. This causes the prefetch code to completely bypass the large IO optimisations for dense areas of metadata. Because there was no obvious change in performance or IO patterns, this wasn't noticed during performance testing.

However, this change mysteriously fixed a regression in xfs/033 in the v3.2.0-alpha release, and this change in behaviour was discovered as part of triaging why it "fixed" the regression. Anyway, restoring the large IO prefetch optimisation results a reapiron a 10 million inode filesystem dropping from 197s to 173s, and the peak IOPS rate in phase 3 dropping from 25,000 to roughly 2,000 by trading off a bandwidth increase of roughly 100% (i.e. 200MB/s to 400MB/s). Phase 4 saw similar changes in IO profile and speed increases.

This, however, re-introduces the regression in xfs/033, which will now be fixed in a separate patch.

---
 libxfs/rdwr.c | 4 ++++
 1 file changed, 4 insertions(+)
libxfs/rdwr.c +4 -0
index ac7739f..78a9b37 100644
@@ -590,6 +590,10 @@ libxfs_getbuf_map(struct xfs_buftarg *btp, struct xfs_buf_map *map,
 	struct xfs_bufkey key = {0};
 	int i;
 
+	if (nmaps == 1)
+		return libxfs_getbuf_flags(btp, map[0].bm_bn, map[0].bm_len,
+					   flags);
+
 	key.buftarg = btp;
 	key.blkno = map[0].bm_bn;
 	for (i = 0; i < nmaps; i++) {
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] libxfs: contiguous buffers are not discontigous

In reply to Dave Chinner
Quoted 25 lines · Dave Chinner

From: Dave Chinner <[email protected]>

When discontiguous directory buffer support was fixed in xfs_repair, (dd9093d xfs_repair: fix discontiguous directory block support) it changed to using libxfs_getbuf_map() to support mapping discontiguous blocks, and the prefetch code special cased such discontiguous buffers.

The issue is that libxfs_getbuf_map() marks all buffers, even contiguous ones - as LIBXFS_B_DISCONTIG, and so the prefetch code was treating every buffer as discontiguous. This causes the prefetch code to completely bypass the large IO optimisations for dense areas of metadata. Because there was no obvious change in performance or IO patterns, this wasn't noticed during performance testing.

However, this change mysteriously fixed a regression in xfs/033 in the v3.2.0-alpha release, and this change in behaviour was discovered as part of triaging why it "fixed" the regression. Anyway, restoring the large IO prefetch optimisation results a reapiron a 10 million inode filesystem dropping from 197s to 173s, and the peak IOPS rate in phase 3 dropping from 25,000 to roughly 2,000 by trading off a bandwidth increase of roughly 100% (i.e. 200MB/s to 400MB/s). Phase 4 saw similar changes in IO profile and speed increases.

Quoted 2 lines

This, however, re-introduces the regression in xfs/033, which will now be fixed in a separate patch.

Thanks for finding this. I was getting close. ;)

It seems fine, although a little unexpected; why do we ever create a map of 1? It feels a little odd to call getbuf_map with only 1 item, and then short-circuit it. Should this be something more obvious in the callers?

Wel, I guess it's pretty much consistent w/ the same behavior in libxfs_readbuf_map()... *shrug*

Quoted 12 lines
---
 libxfs/rdwr.c | 4 ++++
 1 file changed, 4 insertions(+)
libxfs/rdwr.c +0 -0
index ac7739f..78a9b37 100644

libxfs/rdwr.c libxfs_getbuf_map(struct xfs_buftarg *btp, struct xfs_buf_map *map, · line 599

Quoted 10 lines
libxfs/rdwr.c +4 -0
@@ -590,6 +590,10 @@ libxfs_getbuf_map(struct xfs_buftarg *btp, struct xfs_buf_map *map,
 	struct xfs_bufkey key = {0};
 	int i;
 
+	if (nmaps == 1)
+		return libxfs_getbuf_flags(btp, map[0].bm_bn, map[0].bm_len,
+					   flags);
+
 	key.buftarg = btp;
 	key.blkno = map[0].bm_bn;
 	for (i = 0; i < nmaps; i++) {

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

Re: [PATCH 1/2] libxfs: contiguous buffers are not discontigous

Quoted 29 lines · Eric Sandeen
Quoted 25 lines · Dave Chinner

From: Dave Chinner <[email protected]>

When discontiguous directory buffer support was fixed in xfs_repair, (dd9093d xfs_repair: fix discontiguous directory block support) it changed to using libxfs_getbuf_map() to support mapping discontiguous blocks, and the prefetch code special cased such discontiguous buffers.

The issue is that libxfs_getbuf_map() marks all buffers, even contiguous ones - as LIBXFS_B_DISCONTIG, and so the prefetch code was treating every buffer as discontiguous. This causes the prefetch code to completely bypass the large IO optimisations for dense areas of metadata. Because there was no obvious change in performance or IO patterns, this wasn't noticed during performance testing.

However, this change mysteriously fixed a regression in xfs/033 in the v3.2.0-alpha release, and this change in behaviour was discovered as part of triaging why it "fixed" the regression. Anyway, restoring the large IO prefetch optimisation results a reapiron a 10 million inode filesystem dropping from 197s to 173s, and the peak IOPS rate in phase 3 dropping from 25,000 to roughly 2,000 by trading off a bandwidth increase of roughly 100% (i.e. 200MB/s to 400MB/s). Phase 4 saw similar changes in IO profile and speed increases.

Quoted 2 lines

This, however, re-introduces the regression in xfs/033, which will now be fixed in a separate patch.

Quoted 6 lines

Thanks for finding this. I was getting close. ;)

It seems fine, although a little unexpected; why do we ever create a map of 1? It feels a little odd to call getbuf_map with only 1 item, and then short-circuit it. Should this be something more obvious in the callers?

Because we have application code that is building buffer maps from extent maps, and so having a API for both
contigous and discontiguous buffers makes sense. That's the way we do it in the kernel - everything now uses the _map paths - but the userspace code is a much larger surface area to change over and so it's only being done in the places that matter first...

Quoted 2 lines

Wel, I guess it's pretty much consistent w/ the same behavior in libxfs_readbuf_map()... *shrug*

Right. And like I said, it's also how the kernel does stuff, which is why the libxfs_readbuf_map code functions like it does. ;)

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

PATCH 2/2 libxfs: clear stale buffer errors on write

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

If we've read a buffer and it's had an error (e.g a bad CRC) and the caller corrects the problem with the buffer and writes it via libxfs_writebuf() without clearing the error on the buffer, subsequent reads of the buffer while it is still in cache can see that error and fail inappropriately.

xfs/033 demonstrates this error, where phase 3 detects the corrupted root inode and clears, but doesn't clear the b_error field. Later in phase 6, the code that rebuilds the root directory tries to read the root inode and sees a buffer with an error on it, thereby triggering a fatal repair failure:

Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
xfs_inode_buf_verify: XFS_CORRUPTION_ERROR
bad magic number 0x0 on inode 64
....
cleared root inode 64
....
Phase 6 - check inode connectivity... reinitializing root directory
xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.

fatal error -- could not iget root inode -- error - 117 #

Fix this by assuming buffers that are written are clean and correct and hence we can zero the b_error field before retiring the buffer to the cache.

---
 libxfs/rdwr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
libxfs/rdwr.c +10 -0
index 78a9b37..d0ff15b 100644
@@ -890,6 +890,11 @@ libxfs_writebufr(xfs_buf_t *bp)
 int
 libxfs_writebuf_int(xfs_buf_t *bp, int flags)
 {
+	/*
+	 * Clear any error hanging over from reading the buffer. This prevents
+	 * subsequent reads after this write from seeing stale errors.
+	 */
+	bp->b_error = 0;
 	bp->b_flags |= (LIBXFS_B_DIRTY | flags);
 	return 0;
 }
@@ -903,6 +908,11 @@ libxfs_writebuf(xfs_buf_t *bp, int flags)
 			(long long)LIBXFS_BBTOOFF64(bp->b_bn),
 			(long long)bp->b_bn);
 #endif
+	/*
+	 * Clear any error hanging over from reading the buffer. This prevents
+	 * subsequent reads after this write from seeing stale errors.
+	 */
+	bp->b_error = 0;
 	bp->b_flags |= (LIBXFS_B_DIRTY | flags);
 	libxfs_putbuf(bp);
 	return 0;
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] libxfs: clear stale buffer errors on write

In reply to Dave Chinner
Quoted 27 lines · Dave Chinner

From: Dave Chinner <[email protected]>

If we've read a buffer and it's had an error (e.g a bad CRC) and the caller corrects the problem with the buffer and writes it via libxfs_writebuf() without clearing the error on the buffer, subsequent reads of the buffer while it is still in cache can see that error and fail inappropriately.

xfs/033 demonstrates this error, where phase 3 detects the corrupted root inode and clears, but doesn't clear the b_error field. Later in phase 6, the code that rebuilds the root directory tries to read the root inode and sees a buffer with an error on it, thereby triggering a fatal repair failure:

Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
xfs_inode_buf_verify: XFS_CORRUPTION_ERROR
bad magic number 0x0 on inode 64
....
cleared root inode 64
....
Phase 6 - check inode connectivity... reinitializing root directory
xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.

Quoted 6 lines

fatal error -- could not iget root inode -- error - 117 #

Fix this by assuming buffers that are written are clean and correct and hence we can zero the b_error field before retiring the buffer to the cache.

Thanks;

Quoted 25 lines
---
 libxfs/rdwr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
libxfs/rdwr.c +5 -0
index 78a9b37..d0ff15b 100644
@@ -890,6 +890,11 @@ libxfs_writebufr(xfs_buf_t *bp)
 int
 libxfs_writebuf_int(xfs_buf_t *bp, int flags)
 {
+	/*
+	 * Clear any error hanging over from reading the buffer. This prevents
+	 * subsequent reads after this write from seeing stale errors.
+	 */
+	bp->b_error = 0;
 	bp->b_flags |= (LIBXFS_B_DIRTY | flags);
 	return 0;
 }
@@ -903,6 +908,11 @@ libxfs_writebuf(xfs_buf_t *bp, int flags)
 			(long long)LIBXFS_BBTOOFF64(bp->b_bn),

libxfs/rdwr.c libxfs_writebuf(xfs_buf_t *bp, int flags) · line 918

Quoted 10 lines
libxfs/rdwr.c +5 -0
 			(long long)bp->b_bn);
 #endif
+	/*
+	 * Clear any error hanging over from reading the buffer. This prevents
+	 * subsequent reads after this write from seeing stale errors.
+	 */
+	bp->b_error = 0;
 	bp->b_flags |= (LIBXFS_B_DIRTY | flags);
 	libxfs_putbuf(bp);
 	return 0;

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs