catch22: xfs_repair fails, tells me to mount, which fails

Hi,

this morning I woke up to a system without /home, the reason being:

  piper:~# mount /dev/mapper/vg0-home                                       [308]
  mount: Unknown error 990
  piper:~# dmesg | tail -20                                              [32,309]
  Starting XFS recovery on filesystem: dm-0 (logdev: internal)
  Filesystem "dm-0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xffff810028a6bd00, dino bp = 0xffff81002851d140, ino = 33629
  Filesystem "dm-0": XFS internal error xlog_recover_do_inode_trans(1) at line 2352 of file fs/xfs/xfs_log_recover.c.  Caller 0xffffffff88307729

  Call Trace: <ffffffff8830753c>{:xfs:xlog_recover_commit_trans+4198}
        <ffffffff80231360>{submit_bio+184} <ffffffff88307729>{:xfs:xlog_recover_process_data+466}
        <ffffffff883081c6>{:xfs:xlog_do_recovery_pass+585} <ffffffff883096f4>{:xfs:xlog_recover+243}
        <ffffffff88305b2f>{:xfs:xfs_log_mount+1310} <ffffffff8830b671>{:xfs:xfs_mountfs+2116}
        <ffffffff880a0076>{:libata:ata_exec_command+0} <ffffffff8020bcdc>{_atomic_dec_and_lock+57}
        <ffffffff883119ac>{:xfs:xfs_mount+1900} <ffffffff88320bb0>{:xfs:xfs_fs_fill_super+126}
        <ffffffff802bc01d>{get_filesystem+18} <ffffffff802b530c>{sget+850}
        <ffffffff802b4c65>{set_bdev_super+0} <ffffffff802b4c74>{test_bdev_super+0}
        <ffffffff802b5dec>{bd_claim+24} <ffffffff802b5996>{get_sb_bdev+239}
        <ffffffff88320b32>{:xfs:xfs_fs_fill_super+0} <ffffffff802b5605>{do_kern_mount+157}
        <ffffffff802bd86e>{do_mount+1676} <ffffffff8022b2c2>{mntput_no_expire+25}
        <ffffffff802075bb>{find_get_page+33} <ffffffff80212173>{filemap_nopage+387}
        <ffffffff8020878d>{__handle_mm_fault+1276} <ffffffff8020a699>{do_page_fault+1151}
        <ffffffff80249b1f>{sys_mount+138} <ffffffff80259b0e>{system_call+126}
  XFS: log mount/recovery failed: error 990
  XFS: log mount failed

When I try to run xfs_repair, I get:

  piper:~# xfs_repair /dev/mapper/vg0-home                                  [310]
  Phase 1 - find and verify superblock...
  Phase 2 - using internal log
          - zero log...
  ERROR: The filesystem has valuable metadata changes in a log which needs to
  be replayed.  Mount the filesystem to replay the log, and unmount it before
  re-running xfs_repair.  If you are unable to mount the filesystem, then use
  the -L option to destroy the log and attempt a repair.
  Note that destroying the log may cause corruption -- please attempt a mount
  of the filesystem before doing this.

I have already tried the xfs_db method of the "dir2" faq entry, trying to delete that offending inode, but setting core.mode = 0 for that inode did not do anything.

I do not have enough space around to xfsdump those 250Gb, nor would I know how, given that I cannot mount the filesystem and xfsdump works on mounted filesystems only.

I've had really bad experiences with XFS lately [0], but I am not giving up just yet. This time around, I guess I can wait a bit and try to get things fixed.

0. http://blog.madduck.net/geek/2006.08.09-through-with-xfs

Would you have any idea what I should do?

martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: [email protected]
 
"... (ethik und ästhetik sind eins.)"
                                                       -- wittgenstein

FYI, here is how I fixed it, using LVM snapshots to ensure that xfs_repair -L would produce acceptable results:

http://blog.madduck.net/geek/2006.08.30-lvm-for-filesystem-recovery

I'd still be interested to know how I could have gone about fixing it in case I could not have afforded to purge the journal.

martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: [email protected]
 
the reason that every major university
maintains a department of mathematics
is that it's cheaper than
institutionalizing all those people.

Hi Martin,
In this particular case, the log written is corrupted. That's
why, forcing to make the inode going corrupt would not help. The real issue is why log content is not valid.
By any chance, did you saw any disk I/O error ? If there is no
disk I/O error scenario, following information would be more useful.

1. /usr/sbin/xfs_logprint -s 0 -D <device>
2. /usr/sbin/xfs_logprint -d <device>

You can try mounting with -L option but the risk is:
"
ERROR: The filesystem has valuable metadata changes in a log which
needs to
be replayed. Mount the filesystem to replay the log, and unmount it
before
re-running xfs_repair. If you are unable to mount the filesystem,
then use
the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a
mount
of the filesystem before doing this.
"

also sprach Shailendra Tripathi <[email protected]> [2006.08.30.1551 +0200]:

No, definitely not.

Thanks for the info. I managed to restore the filesystem with xfs_repair -L in the mean time (see followup post), but this is what I'll keep in mind for the next time.

Btw: there's the xfs and linux-xfs mailing list, and I am confused about whether they're separate or merged, and which one I should be using. If you have any input, I'd love to hear it.

martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: [email protected]
 
no micro$oft components were used
in the creation or posting of this email.
therefore, it is 100% virus free
and does not use html by default (yuck!).

xfs@ is the current one, linux-xfs@ is the old one, and is aliased to xfs@ (there was probably a post about this when xfs@ was created).

-Eric