repair: fixes for 3.2
cover repair: fixes for 3.2
Hi Arkadiusz,
These are the fixes I have so far from working through the metadumps you supplied me with. The unobfuscated metadump repairs cleanly with these patches, but the obfuscated one still has a directory rebuild issue that I haven't got to the bottom of yet. That results in an error in phase 6 like:
Metadata corruption detected at block 0x2af84770/0x1000
name create failed in ino 2306912354 (117), filesystem may be out of space
It appears that a read verifier is on a recently created directory leaf block and that is failing. I'm not yet sure why the leaf block is corrupt, nor why the verifier is even being run on it seeing as it was only allocated and initialised during the directory rebuild. That directory rebuil dfailure is the reason for all the disconected inode that end up in lost+found, and I think it's the only remaining issue that I need to solve.
Can you test the patches on you machine and see if you get the same results?
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/4 repair: handle directory block corruption in phase 6
From: Dave Chinner <[email protected]>
This should only occur in no-modify mode, but when we fail to find the last extent in a directory btree due to corruption we need to trash the directory if it's the first data block we find the error on. That is because there is nothing to recover from the directory, and if we try to scan it xfs_reapir segv's because nothing has been read from disk.
Also catch a memory allocation failure in this code, too.
--- repair/phase6.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
repair/phase6.c +17 -2
@@ -2179,7 +2179,7 @@ longform_dir2_entry_check(xfs_mount_t *mp, freetab =malloc (FREETAB_SIZE (ip->i_d.di_size / mp->m_dirblksize));if (!freetab) {do_error ( -_ ("malloc failed in longform_dir2_entry_check (%" PRId64" bytes)\n" ), +_ ("malloc failed in longform_dir2_entry_check (%" PRId64" bytes)\n" ),FREETAB_SIZE (ip->i_d.di_size / mp->m_dirblksize));exit (1 ); }
@@ -2191,6 +2191,11 @@ longform_dir2_entry_check(xfs_mount_t *mp, } num_bps = freetab->naents; bplist =calloc (num_bps,sizeof (struct xfs_buf *)); +if (!bplist) +do_error ( +_ ("calloc failed in longform_dir2_entry_check (%zu bytes)\n" ), + num_bps *sizeof (struct xfs_buf *)); +/* is this a block, leaf, or node directory? */ libxfs_dir2_isblock (NULL , ip, &isblock);libxfs_dir2_isleaf (NULL , ip, &isleaf);
@@ -2203,8 +2208,18 @@ longform_dir2_entry_check(xfs_mount_t *mp,int error; next_da_bno = da_bno + mp->m_dirblkfsbs -1 ; -if (bmap_next_offset (NULL , ip, &next_da_bno,XFS_DATA_FORK )) +if (bmap_next_offset (NULL , ip, &next_da_bno,XFS_DATA_FORK )) { +/* +* if this is the first block, there isn't anything we +* can recover so we just trash it. +*/ +if (da_bno ==0 ) { + fixit++; + goto out_fix; + }break ; + } + db =xfs_dir2_da_to_db (mp, da_bno);if (db >= num_bps) {/* more data blocks than expected */
Signature · 6 lines
2.0.0 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/4 libxfs: buffers aren't stale once written
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When rebuilding a bad directory, repair first truncates away all the blocks in the directory. This removes blocks from the bmap btree, and when those blocks are freed the bmap btree code invalidates them. This marks the buffers LIBXFS_B_STALE so that we don't try to write stale data from that buffer at a later time.
However, when rebuilding the directory, the block gets reallocated and new metadata gets written into it. however, it is still marked stale and hence the new, corrected metadata is never written. Hence if we write new metadata to a stale buffer, we have to clear the LIBXFS_B_STALE flag to ensure that the new data gets written.
Note that this can affect more than just bmap btree buffers - there are several different places in the libxfs code that mark buffers stale via xfs_trans_binval() during repair operations.
--- libxfs/rdwr.c | 2 ++ 1 file changed, 2 insertions(+)
libxfs/rdwr.c +2 -0
@@ -928,6 +928,7 @@ libxfs_writebuf_int(xfs_buf_t *bp, int flags)* subsequent reads after this write from seeing stale errors. */ bp->b_error =0 ; + bp->b_flags &= ~LIBXFS_B_STALE ; bp->b_flags |= (LIBXFS_B_DIRTY | flags);return 0 ; }
@@ -946,6 +947,7 @@ libxfs_writebuf(xfs_buf_t *bp, int flags)* subsequent reads after this write from seeing stale errors. */ bp->b_error =0 ; + bp->b_flags &= ~LIBXFS_B_STALE ; bp->b_flags |= (LIBXFS_B_DIRTY | flags);libxfs_putbuf (bp);return 0 ;
Signature · 6 lines
2.0.0 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/4 repair: fix quota inode handling in secondary superblocks
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Changes to support separate project quota inodes changed the way quota inodes got written to the superblock. The current code is tailored for the needs to the kernel, where the inodes should only be written if certain falgs are set saying a quota type is enabled.
Unfortunately, when recovering a corrupt secondary superblock, we need to unconditionally write the quota inode fields after we unconditionally zero the quota flags field. The result of this bug is that the bad quota inode fields cannot be cleared and hence always are reported by bad by repair in subsequent runs.
Fix this by directly clearing the quota inodes in the superblock buffers so that we do need to set special flags to get xfs_sb_to_disk() to do the right thing as setting flags leave bad flag values in the superblock instead of bad inode numbers....
Also, when clearing the inode numbers, write them as NULLFSINO rather than 0 as this is what the kernel will write them as if quota is turned off.
--- include/libxfs.h | 1 + libxfs/rdwr.c | 4 ++-- repair/agheader.c | 52 ++++++++++++++++++++++++++++++++++------------------ repair/sb.c | 2 ++ repair/scan.c | 1 + repair/xfs_repair.c | 1 + 6 files changed, 41 insertions(+), 20 deletions(-)
include/libxfs.h +1 -0
@@ -759,6 +759,7 @@ bool xfs_dinode_verify(struct xfs_mount *mp, xfs_ino_t ino,/* xfs_sb.h */ #define libxfs_mod_sb xfs_mod_sb#define libxfs_sb_from_disk xfs_sb_from_disk +#define libxfs_sb_quota_from_disk xfs_sb_quota_from_disk#define libxfs_sb_to_disk xfs_sb_to_disk/* xfs_symlink.h */
libxfs/rdwr.c +2 -2
@@ -907,10 +907,10 @@ libxfs_writebufr(xfs_buf_t *bp) }#ifdef IO_DEBUG -printf ("%lx: %s: wrote %u bytes, blkno=%llu(%llu), %p\n" , +printf ("%lx: %s: wrote %u bytes, blkno=%llu(%llu), %p, error %d\n" ,pthread_self (), __FUNCTION__, bp->b_bcount, (long long )LIBXFS_BBTOOFF64 (bp->b_bn), - (long long )bp->b_bn, bp); + (long long )bp->b_bn, bp, error);#endif if (!error) { bp->b_flags |=LIBXFS_B_UPTODATE ;
repair/agheader.c +34 -18
79 lines not on this page · raw message
repair/sb.c +2 -0
@@ -138,6 +138,7 @@ find_secondary_sb(xfs_sb_t *rsb)for (i =0 ; !done && i < bsize; i +=BBSIZE ) { c_bufsb = (char *)sb + i;libxfs_sb_from_disk (&bufsb, (xfs_dsb_t *)c_bufsb); +libxfs_sb_quota_from_disk (&bufsb);if (verify_sb (c_bufsb, &bufsb,0 ) !=XR_OK )continue ;
@@ -538,6 +539,7 @@ get_sb(xfs_sb_t *sbp, xfs_off_t off, int size, xfs_agnumber_t agno)do_error ("%s\n" ,strerror (error)); }libxfs_sb_from_disk (sbp, buf); +libxfs_sb_quota_from_disk (sbp); rval =verify_sb ((char *)buf, sbp, agno ==0 );free (buf);
repair/scan.c +1 -0
@@ -1496,6 +1496,7 @@ scan_ag( goto out_free_sb; }libxfs_sb_from_disk (sb,XFS_BUF_TO_SBP (sbbuf)); +libxfs_sb_quota_from_disk (sb); agfbuf = libxfs_readbuf(mp->m_dev, XFS_AG_DADDR(mp, agno, XFS_AGF_DADDR(mp)),
repair/xfs_repair.c +1 -0
@@ -782,6 +782,7 @@ main(int argc, char **argv)/* make sure the per-ag freespace maps are ok so we can mount the fs */ phase2 (mp, phase2_threads);timestamp (PHASE_END ,2 ,NULL ); +libxfs_bcache_flush ();if (do_prefetch)init_prefetch (mp);
Signature · 6 lines
2.0.0 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/4 repair: get rid of BADFSINO
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When we find a bad dirent, we "clear" the inode the inode number by writing BADFSINO to the inode number in the entry:
#define BADFSINO ((xfs_ino_t)0xfeffffffffffffffULL)
We then capture this bad inode number later in the same function either in the same pass or in a later phase and junk the entry. When we junk the entry, we write a "/" over the first character of the dirent name, which is then detected up later by the directory rebuild and ignored.
The issue with this is that the directory buffer can be written to disk between the dirent being marked with BADFSINO and the directory rebuild processing in phase 6, resulting in the directory block verifier firing this error:
Invalid inode number 0xfeffffffffffffffxfs_dir_ino_validate: XFS_ERROR_REPORT
Metadata corruption detected at block 0x11fbb698/0x1000libxfs_writebufr: write verifer failed on bno 0x11fbb698/0x1000
And so will not write the *corrupt block* to disk. The result is that we don't repair a corruption in the directory block correctly and subsequent repair runs continue to find problems with the directory.
We really don't need both BADFSINO *and* overwriting the dirent name with "/" to mark an entry as junked. They both mean exactly the same thing, so get rid of BADFSINO and only use the name junking to mark dirents as bad. This prevents the directory data block verifier from triggering on bad inode numbers, and so the later reread of the block will find the junked entries correctly.
--- repair/dir2.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-)
repair/dir2.c +5 -13
@@ -28,13 +28,6 @@#include "progress.h" /* -* Tag bad directory entries with this. -* We can't tag them with -1 since that will look like a -* data_unused_t instead of a data_entry_t. -*/ -#define BADFSINO ((xfs_ino_t)0xfeffffffffffffffULL) - -/* * Known bad inode list. These are seen when the leaf and node * block linkages are incorrect. */
@@ -1314,7 +1307,7 @@ process_dir2_data(* Conditions must either set clearino to zero or set * clearreason why it's being cleared. */ -if (!ino_discovery && ent_ino ==BADFSINO ) { +if (!ino_discovery && dep->name[0 ] =='/' ) {/* * Don't do a damned thing. We already found this * (or did it ourselves) during phase 3.
@@ -1401,8 +1394,7 @@ _("entry at block %u offset %" PRIdPTR " in directory inode %" PRIu64do_warn (_ ("\tclearing inode number in entry at offset %" PRIdPTR"...\n" ), (intptr_t )ptr - (intptr_t )d); - dep->inumber =cpu_to_be64 (BADFSINO ); - ent_ino =BADFSINO ; + dep->name[0 ] ='/' ; *dirty =1 ; }else { do_warn(
@@ -1415,7 +1407,7 @@ _("\twould clear inode number in entry at offset %" PRIdPTR "...\n"),* discovery is turned on). Otherwise, we'd complain a lot * during phase 4. */ - junkit = ent_ino ==BADFSINO ; + junkit = dep->name[0 ] =='/' ; nm_illegal =namecheck ((char *)dep->name, dep->namelen);if (ino_discovery && nm_illegal) { do_warn(
@@ -1427,11 +1419,11 @@ _("entry at block %u offset %" PRIdPTR " in directory inode %" PRIu64 " has ille/* * Now we can mark entries with BADFSINO's bad. */ -if (!no_modify && ent_ino ==BADFSINO ) { - dep->name[0 ] ='/' ; +if (!no_modify && dep->name[0 ] =='/' ) { *dirty =1 ; junkit =0 ; } +/* * Special .. entry processing. */
Signature · 6 lines
2.0.0 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] repair: get rid of BADFSINO
Quoted 15 lines · Dave Chinner
From: Dave Chinner <[email protected]>
so get rid of BADFSINO and only use the name junking to mark dirents as bad. This prevents the directory data block verifier from triggering on bad inode numbers, and so the later reread of the block will find the junked entries correctly.
Signed-off-by Dave Chinner <[email protected]>--- repair/dir2.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-)(quoted diff) +0
-0@@ -1427,11 +1419,11 @@ _("entry at block %u offset %" PRIdPTR " indirectory inode %" PRIu64 " has ille /*
repair/dir2.c _("entry at block %u offset %" PRIdPTR " in directory inode %" PRIu64 " has ille · line 1421
Quoted 2 lines
repair/dir2.c +0
-0* Now we can mark entries with BADFSINO's bad. */
Comment above needs update.
Quoted 10 lines
repair/dir2.c +2
-2-if (!no_modify && ent_ino ==BADFSINO ) { - dep->name[0 ] ='/' ; +if (!no_modify && dep->name[0 ] =='/' ) { *dirty =1 ; junkit =0 ; } +/* * Special .. entry processing. */
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 19 lines · Dave Chinner
Hi Arkadiusz,
These are the fixes I have so far from working through the metadumps you supplied me with. The unobfuscated metadump repairs cleanly with these patches, but the obfuscated one still has a directory rebuild issue that I haven't got to the bottom of yet. That results in an error in phase 6 like:
Metadata corruption detected at block 0x2af84770/0x1000
name create failed in ino 2306912354 (117), filesystem may be out of spaceIt appears that a read verifier is on a recently created directory leaf block and that is failing. I'm not yet sure why the leaf block is corrupt, nor why the verifier is even being run on it seeing as it was only allocated and initialised during the directory rebuild. That directory rebuil dfailure is the reason for all the disconected inode that end up in lost+found, and I think it's the only remaining issue that I need to solve.
Quoted 2 lines
Can you test the patches on you machine and see if you get the same results?
Testing on non obfuscated image here.
Most of issues got fixed, first xfs_repair run:
http://ixion.pld-linux.org/~arekm/p2/x1/repair-3.2-dchinner1-4patches-run1.txt
second xfs_repair run (reports no problems):
http://ixion.pld-linux.org/~arekm/p2/x1/repair-3.2-dchinner1-4patches-run2.txt
When trying to mount
"XFS (loop0): Failed to initialize disk quotas."
so this issue left.
Question:
Phase 2 - using internal log
- zero log...zero_log: head block 2 tail block 2
- scan filesystem freespace and inode maps...
Metadata CRC error detected at block 0x0/0x200
Is "Metadata CRC error detected at block" expected here? I mean v4 fs, so no CRC.
Thanks a lot for the fixes.
Quoted 3 lines
Cheers,
Dave.
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 28 lines · Arkadiusz Miśkiewicz
Quoted 19 lines · Dave Chinner
Hi Arkadiusz,
These are the fixes I have so far from working through the metadumps you supplied me with. The unobfuscated metadump repairs cleanly with these patches, but the obfuscated one still has a directory rebuild issue that I haven't got to the bottom of yet. That results in an error in phase 6 like:
Metadata corruption detected at block 0x2af84770/0x1000
name create failed in ino 2306912354 (117), filesystem may be out of spaceIt appears that a read verifier is on a recently created directory leaf block and that is failing. I'm not yet sure why the leaf block is corrupt, nor why the verifier is even being run on it seeing as it was only allocated and initialised during the directory rebuild. That directory rebuil dfailure is the reason for all the disconected inode that end up in lost+found, and I think it's the only remaining issue that I need to solve.
Quoted 2 lines
Can you test the patches on you machine and see if you get the same results?
Testing on non obfuscated image here.
Most of issues got fixed, first
xfs_repairrun:
http://ixion.pld-linux.org/~arekm/p2/x1/repair-3.2-dchinner1-4patches-run1.txt
Quoted 6 lines
second
xfs_repairrun (reports no problems):
http://ixion.pld-linux.org/~arekm/p2/x1/repair-3.2-dchinner1-4patches-run2.txtWhen trying to mount
"XFS (loop0): Failed to initialize disk quotas."
so this issue left.
Yeah, I haven't got that far yet....
Quoted 10 lines
Question:
Phase 2 - using internal log
- zero log...zero_log: head block 2 tail block 2
- scan filesystem freespace and inode maps...
Metadata CRC error detected at block 0x0/0x200Is "Metadata CRC error detected at block" expected here? I mean v4 fs, so no CRC.
Given that it was followed by:
zeroing unused portion of primary superblock (AG #0)
Then there was garbage in the superblock that made the verifier think that maybe it was missing a feature bit. It didn't come up the second time, so everything is fine....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs