Skip to contentSkip to conversation

NFS && loop == VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...

PATCH Nathaniel Stahl 2 participants 2 messages Mar 15, 2005

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

On the 2.6 kernel, using NFS and the loopback device, I can trivially cause this to happen. Create a container file on an NFS imported directory. Make a filesystem in the container file, and loopback mount it. Write another file onto that newly mounted filesystem, then unmount it and the original NFS filesystem that held the container. Usually, the error will occur.

(Scripts and patches I've tried are reproduced at the end of the message)

The call path when the error occurs is:
sys_umount->_mntput->__mntput->deactivate_super->nfs_kill_super->

       kill_anon_super->generic_shutdown_super

nfs_kill_super calls kill_anon_super - which calls generic_shutdown_super, which outputs the message.

My guess was that some of the inodes are still being referenced by outstanding RPC operations. Putting a call to 'rpc_show_tasks()' immediately after the call to kill_anon_super in nfs_kill_super produces output like:

(with nfsv3)
VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day... -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- --exit-- 11911 0021 0401 000000 de2b2e00 100003 cf251000 00060000 xprt_pending e0a65dc4 e0aa4ee8

From kallsyms: e0a65dc4 == call_status, e0aa4ee8 = nfs3_commit_done

nfs_commit_done calls nfs_inode_remove_request, which can release an inode reference...

When a file is closed (via sys_close) it calls the filesystem flush operation. The loop device doesn't do this. If I update loop.c to call the f_op->flush on the file pointer prior to the file backing the loopback device being released, the VFS error doesn't occur (at least not trivially with those scripts on 2.6.11.3). mmaps will also call fput without calling flush, but I can't seem to reproduce the problem this way, so maybe that's a red herring. Changing the loop device to call sync_page_range prior to releasing the file also seems to solve the problem. Should one or both of these be done? (Perhaps neither?)

Any ideas would be appreciated.

############################################################

Without any patches, changing the NFS mount options to 'sync' - in a more complicated senario using our control software and not the simple scripts generates errors like:

kernel: Buffer I/O error on device loop1, logical block 2368

Mar 11 13:45:12 sandevel kernel: lost page write due to I/O error on loop1

I haven't been able to reproduce this using my simpler scripts.

(nfsvers=2/sync full system)
Buffer I/O error on device loop1, logical block 2396
lost page write due to I/O error on loop1
-pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- --exit--

03133 0004 0080 000000 de15e400 100003 d7501000 00060000 xprt_pending e0a65dc4        0
03182 0008 0080 000000 de16de00 100003 d36d40ac 00060000 xprt_pending e0a65dc4        0
03183 0008 0080 000000 d2c22a00 100003 dd66a60c 00060000 xprt_pending e0a65dc4        0

############################################################

# With an NFS filesystem exported from $SERVERDIR to the client # This usually causes the problem to occur.

SERVERDIR=172.31.37.2:/mnt/test
NFSDIR=/mnt/test
LOOPDIR=/mnt/loop

mkdir -p $NFSDIR $LOOPDIR
mount -t nfs $SERVERDIR $NFSDIR
dd if=/dev/zero of=$NFSDIR/filesystem bs=1024 seek=102400 count=1 mke2fs -F $NFSDIR/filesystem
mount -o loop $NFSDIR/filesystem $LOOPDIR
dd if=/dev/zero of=$LOOPDIR/foo bs=1024 count=1000 umount $LOOPDIR
umount $NFSDIR

############################################################

# This one is slightly more likely to generate the error.

SERVERDIR=172.31.37.2:/mnt/test
SERVERDIR2=172.31.37.2:/mnt/test2
NFSDIR=/mnt/test
NFSDIR2=/mnt/test2
LOOPDIR=/mnt/loop
LOOPDIR2=/mnt/loop2

mkdir -p $NFSDIR $NFSDIR2 $LOOPDIR $LOOPDIR2
mount -t nfs -o nfsvers=2 $SERVERDIR $NFSDIR
mount -t nfs -o nfsvers=2 $SERVERDIR2 $NFSDIR2
dd if=/dev/zero of=$NFSDIR/filesystem bs=1024 seek=102400 count=1 dd if=/dev/zero of=$NFSDIR2/filesystem bs=1024 seek=102400 count=1 mke2fs -F $NFSDIR/filesystem
mke2fs -F $NFSDIR2/filesystem
mount -o loop $NFSDIR/filesystem $LOOPDIR
mount -o loop $NFSDIR2/filesystem $LOOPDIR2
dd if=/dev/zero of=$LOOPDIR/foo bs=1024 count=1000
dd if=/dev/zero of=$LOOPDIR2/foo bs=1024 count=1000 umount $LOOPDIR
umount $NFSDIR
umount $LOOPDIR2
umount $NFSDIR2

drivers/block/loop.c renamed from drivers/block/loop.c.orig +10 -0
@@ -608,6 +608,12 @@
 	if (get_loop_size(lo, file) != get_loop_size(lo, old_file))
 		goto out_putf;
 
+	if (old_file->f_op && old_file->f_op->flush) {
+		error = old_file->f_op->flush(old_file);
+		if (error)
+			goto out_putf;
+	}
+
 	/* and ... switch */
 	error = loop_switch(lo, file);
 	if (error)
@@ -820,6 +826,10 @@
 	bd_set_size(bdev, 0);
 	mapping_set_gfp_mask(filp->f_mapping, gfp);
 	lo->lo_state = Lo_unbound;
+
+	if (filp->f_op && filp->f_op->flush)
+		filp->f_op->flush(filp);
+
 	fput(filp);
 	/* This is safe: open() is still holding a reference. */
 	module_put(THIS_MODULE);

610a611,626

Quoted 16 lines

/*

	if (old_file->f_op && old_file->f_op->flush) {
		error = old_file->f_op->flush(old_file);
		if (error)
			goto out_putf;
	}
*/
	error = sync_page_range(old_file->f_dentry->d_inode,
		old_file->f_dentry->d_inode->i_mapping, 0,
		i_size_read(old_file->f_dentry->d_inode));
	if (error) {
		char b[BDEVNAME_SIZE];
		printk(KERN_WARNING "Failed to sync all pages for old backing "
			"file of loop device %s.\n", bdevname(bdev,b));
	}

785a802

Quoted 1 line
	int err;

803a821,829

Quoted 9 lines
	err = sync_page_range(filp->f_dentry->d_inode,
		filp->f_dentry->d_inode->i_mapping, 0,
		i_size_read(filp->f_dentry->d_inode));
	if (err) {
		char b[BDEVNAME_SIZE];
		printk(KERN_WARNING "Failed to sync all pages for old backing "
			"file of loop device %s.\n", bdevname(bdev,b));
	}

822a849,854

Quoted 6 lines

/*

	if (filp->f_op && filp->f_op->flush)
		filp->f_op->flush(filp);

*/

#1 lore raw

må den 14.03.2005 Klokka 18:22 (-0800) skreiv Nathaniel Stahl:

Quoted 13 lines

On the 2.6 kernel, using NFS and the loopback device, I can trivially cause this to happen. Create a container file on an NFS imported directory. Make a filesystem in the container file, and loopback mount it. Write another file onto that newly mounted filesystem, then unmount it and the original NFS filesystem that held the container. Usually, the error will occur.

(Scripts and patches I've tried are reproduced at the end of the message)

The call path when the error occurs is:
sys_umount->_mntput->__mntput->deactivate_super->nfs_kill_super->

        kill_anon_super->generic_shutdown_super
Quoted 7 lines

nfs_kill_super calls kill_anon_super - which calls generic_shutdown_super, which outputs the message.

My guess was that some of the inodes are still being referenced by outstanding RPC operations. Putting a call to 'rpc_show_tasks()' immediately after the call to kill_anon_super in nfs_kill_super produces output like:

Coincidentally, I just got a similar bugreport. Could you try out the following patch to the NFS client?

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>

NFS: Ensure that dirty pages are written with the right creds.

When doing shared mmap writes, the resulting dirty NFS pages may find themselves incapable of being flushed out if I/O is started after the file was released.
Make sure we start I/O on all existing dirty pages in nfs_file_release().

---
 file.c |    3 +++
 1 files changed, 3 insertions(+)
fs/nfs/file.c +3 -0
Index: linux-2.6.11-up/fs/nfs/file.c
===================================================================
@@ -108,6 +108,9 @@ nfs_file_open(struct inode *inode, struc
 static int
 nfs_file_release(struct inode *inode, struct file *filp)
 {
+	/* Ensure that dirty pages are flushed out with the right creds */
+	if (filp->f_mode & FMODE_WRITE)
+		filemap_fdatawrite(filp->f_mapping);
 	return NFS_PROTO(inode)->file_release(inode, filp);
 }