Calltrace in 2.6.27 kernel]

Quoted 9 lines

Hi

I tested kernel 2.6.27 with stress test using fsstress,dd,LVM and snapshots. After a few hours and overflow of snapshot I got call trace:

device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.

This is a message from device mapper telling your that it got an EIO in pending_complete()

Quoted 6 lines

I/O error in filesystem ("dm-49") meta-data dev dm-49 block
0x1d4c3b0 ("xlog_recover_do..(read#2)")
error 5 buf count 8192
XFS: log mount/recovery failed: error 5
XFS: log mount failed

And now XFS complains that it got this error passed up, so far so good.

Quoted 4 lines

BUG: unable to handle kernel NULL pointer dereference at 0000002c
IP: [<c029553d>] xfs_log_force_umount+0x3d/0x170

*pdpt = 00000000345cf001 *pde = 0000000000000000

But ut should not crash. Can you run

	addr2line -e /path/to/your/kernel xfs_log_force_umount+0x3d/0x170

(you'll need a kernel with debuginfo for that)

Hi

I ran addr2line for both vmlinux and for first call trace (2.6.27) add2line returned:

addr2line -e ./vmlinux c029553d
fs/xfs/xfs_log.c:3528

in source:

        /*
         * If this happens during log recovery, don't worry about
         * locking; the log isn't open for business yet.
         */
        if (!log ||
            log->l_flags & XLOG_ACTIVE_RECOVERY) {
                mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
                XFS_BUF_DONE(mp->m_sb_bp);                            

//3528 line

Quoted 18 lines · On Fri, Nov 14, 2008 at 09:49:47AM +0100, [email protected] wrote:

Hi

I ran addr2line for both vmlinux and for first call trace (2.6.27) add2line returned:

addr2line -e ./vmlinux c029553d
fs/xfs/xfs_log.c:3528

in source:

        /*
         * If this happens during log recovery, don't worry about
         * locking; the log isn't open for business yet.
         */
        if (!log ||
            log->l_flags & XLOG_ACTIVE_RECOVERY) {
                mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
                XFS_BUF_DONE(mp->m_sb_bp);                            

Quoted 2 lines

//3528 line

Thanks a lot. I think I figured out what happens:

- we fail the buffer read in xlog_recover_do_inode_trans, and thus abort the mount.
- but before that log recovery has started delayed writeback of inode buffers
- during the mount error handling we call xfs_freesb, which NULLs mp->m_sb_bp after the inode has been unmounted
- but the device close tries to flush all delayed buffers after that, and the inode buffer triggers the filesystem shutdown case because it also had an I/O error.
- Now we try to mark the superblock buffer done, but it's not there anymore.

The easy fix is to just check for a NULL mp->m_sb_bp before marking it done. Note that this will only fix the XFS oops, but not the underling DM issue, for which I'd ask on the device mapper list.

Index: xfs-2.6/fs/xfs/xfs_log.c
===================================================================

fs/xfs/xfs_log.c +2 -1

@@ -3525,7 +3525,8 @@ xfs_log_force_umount(
 	if (!log ||
 	    log->l_flags & XLOG_ACTIVE_RECOVERY) {
 		mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
-		XFS_BUF_DONE(mp->m_sb_bp);
+		if (mp->m_sb_bp)
+			XFS_BUF_DONE(mp->m_sb_bp);
 		return 0;
 	}