nfsdv4 leaks file descriptors

DISCUSSION Gerhard Heift 2 participants 8 messages Apr 12, 2011 – Apr 20, 2011

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Gerhard Heift

author

2011-04-12 14:31 UTC #0 lore raw

Hello,

I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:

/data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)

On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the directory under /media(nfs4import with

# mount -t nfs4 ip:/ /media/nfs4import

After a lot running fdupes on this folder the server kernel starts throwing messages like this one:

VFS: file-max limit [number] reached

After a reboot I recognised that the number of used file descriptors increases during the run of fdupes (see /proc/sys/fs/file-nr). Even after unmounting the exported filesystem on the client, unexport the file system on the server and stopping the nfs-server the file descriptors are still allocated and unmounting the filesystem is not possible.

After a little bit of research I could create a small program, which increases the used file descriptors by one.

Regards,
Gerhard

#include <stdio.h>

#define DIR "/media/nfs4import/"

int main(void) {
	FILE *f1;
	FILE *f2;
	char b[1024];
	int i;

	// create file1
	f1 = fopen(DIR "file1.txt", "w");
	fwrite("test\n", 5, 1, f1);
	close(f1);

	// open file1 in read mode
	f1 = fopen(DIR "file1.txt", "r");

	for (i = 0; i < 1; i++) {
		// open file2 in write mode and close it
		f2 = fopen(DIR "file2.txt", "w");
		fclose(f2);
	
		// open file2 in read mode and close it
		// this causes the leak of a file descriptor on the server
		// just watch /proc/sys/fs/file-nr on the server
		f2 = fopen(DIR "file2.txt", "r");
		fclose(f2);
	}
	fclose(f1);

	return 0;
}

J. Bruce Fields

2011-04-13 21:12 UTC #1 lore raw

Quoted 4 lines · Gerhard Heift

Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:

Should be fixed in more recent stable kernel; let me know if not.

--b.

Quoted 59 lines

/data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the directory under /media(nfs4import with
# mount -t nfs4 ip:/ /media/nfs4import
After a lot running fdupes on this folder the server kernel starts throwing messages like this one:
VFS: file-max limit [number] reached
After a reboot I recognised that the number of used file descriptors increases during the run of fdupes (see /proc/sys/fs/file-nr). Even after unmounting the exported filesystem on the client, unexport the file system on the server and stopping the nfs-server the file descriptors are still allocated and unmounting the filesystem is not possible.
After a little bit of research I could create a small program, which increases the used file descriptors by one.
Regards,
Gerhard
#include <stdio.h>
#define DIR "/media/nfs4import/"
int main(void) {
	FILE *f1;
	FILE *f2;
	char b[1024];
	int i;

	// create file1
	f1 = fopen(DIR "file1.txt", "w");
	fwrite("test\n", 5, 1, f1);
	close(f1);

	// open file1 in read mode
	f1 = fopen(DIR "file1.txt", "r");

	for (i = 0; i < 1; i++) {
		// open file2 in write mode and close it
		f2 = fopen(DIR "file2.txt", "w");
		fclose(f2);
	
		// open file2 in read mode and close it
		// this causes the leak of a file descriptor on the server
		// just watch /proc/sys/fs/file-nr on the server
		f2 = fopen(DIR "file2.txt", "r");
		fclose(f2);
	}
	fclose(f1);

	return 0;
}

Gerhard Heift

author

2011-04-14 05:48 UTC #2 lore raw

Quoted 7 lines · J. Bruce Fields

Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:
Should be fixed in more recent stable kernel; let me know if not.

An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.

Quoted 1 line

--b.

Regards,
Gerhard

Quoted 61 lines

Quoted 58 lines
/data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the directory under /media(nfs4import with
# mount -t nfs4 ip:/ /media/nfs4import
After a lot running fdupes on this folder the server kernel starts throwing messages like this one:
VFS: file-max limit [number] reached
After a reboot I recognised that the number of used file descriptors increases during the run of fdupes (see /proc/sys/fs/file-nr). Even after unmounting the exported filesystem on the client, unexport the file system on the server and stopping the nfs-server the file descriptors are still allocated and unmounting the filesystem is not possible.
After a little bit of research I could create a small program, which increases the used file descriptors by one.
Regards,
Gerhard
#include <stdio.h>
#define DIR "/media/nfs4import/"
int main(void) {
	FILE *f1;
	FILE *f2;
	char b[1024];
	int i;

	// create file1
	f1 = fopen(DIR "file1.txt", "w");
	fwrite("test\n", 5, 1, f1);
	close(f1);

	// open file1 in read mode
	f1 = fopen(DIR "file1.txt", "r");

	for (i = 0; i < 1; i++) {
		// open file2 in write mode and close it
		f2 = fopen(DIR "file2.txt", "w");
		fclose(f2);
	
		// open file2 in read mode and close it
		// this causes the leak of a file descriptor on the server
		// just watch /proc/sys/fs/file-nr on the server
		f2 = fopen(DIR "file2.txt", "r");
		fclose(f2);
	}
	fclose(f1);

	return 0;
}

J. Bruce Fields

2011-04-14 19:10 UTC #3 lore raw

Quoted 11 lines · Gerhard Heift

Quoted 7 lines · J. Bruce Fields
Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:
Should be fixed in more recent stable kernel; let me know if not.
An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.

Ugh. OK, investigating.

--b.

Quoted 67 lines

Quoted 2 lines
--b.
Regards,
Gerhard
Quoted 61 lines
Quoted 58 lines
/data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the directory under /media(nfs4import with
# mount -t nfs4 ip:/ /media/nfs4import
After a lot running fdupes on this folder the server kernel starts throwing messages like this one:
VFS: file-max limit [number] reached
After a reboot I recognised that the number of used file descriptors increases during the run of fdupes (see /proc/sys/fs/file-nr). Even after unmounting the exported filesystem on the client, unexport the file system on the server and stopping the nfs-server the file descriptors are still allocated and unmounting the filesystem is not possible.
After a little bit of research I could create a small program, which increases the used file descriptors by one.
Regards,
Gerhard
#include <stdio.h>
#define DIR "/media/nfs4import/"
int main(void) {
	FILE *f1;
	FILE *f2;
	char b[1024];
	int i;

	// create file1
	f1 = fopen(DIR "file1.txt", "w");
	fwrite("test\n", 5, 1, f1);
	close(f1);

	// open file1 in read mode
	f1 = fopen(DIR "file1.txt", "r");

	for (i = 0; i < 1; i++) {
		// open file2 in write mode and close it
		f2 = fopen(DIR "file2.txt", "w");
		fclose(f2);
	
		// open file2 in read mode and close it
		// this causes the leak of a file descriptor on the server
		// just watch /proc/sys/fs/file-nr on the server
		f2 = fopen(DIR "file2.txt", "r");
		fclose(f2);
	}
	fclose(f1);

	return 0;
}

Gerhard Heift

author

2011-04-15 13:20 UTC #4 lore raw

Quoted 8 lines · J. Bruce Fields

Quoted 7 lines · Gerhard Heift
Quoted 6 lines · J. Bruce Fields
Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:

Quoted 6 lines

Quoted 4 lines
Quoted 1 line
Should be fixed in more recent stable kernel; let me know if not.
An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.
Ugh. OK, investigating.

It took me a bit of time, but I located the commit which introduced this
behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49

Quoted 1 line

--b.

Regards,
Gerhard

J. Bruce Fields

2011-04-15 22:36 UTC #5 lore raw

Quoted 11 lines · Gerhard Heift

Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · Gerhard Heift
Quoted 7 lines · J. Bruce Fields
Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:
Should be fixed in more recent stable kernel; let me know if not.

Quoted 7 lines

Quoted 4 lines
Quoted 2 lines
An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.
Ugh. OK, investigating.
It took me a bit of time, but I located the commit which introduced this
behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49

Doh--probably we just need this (lightly tested).

Thanks for tracking this down.

--b.

commit 21cd8aaf5efb086361cf12f9245c14a3b19fbafd

Author: J. Bruce Fields <[email protected]>
Date:   Fri Apr 15 18:08:26 2011 -0400

nfsd4: fix struct file leak on delegation

Introduced by acfdf5c383b38f7f4dddae41b97c97f1ae058f49.

Reported-by Gerhard Heift <[email protected]>

Signed-off-by J. Bruce Fields <[email protected]>

fs/nfsd/nfs4state.c +1 -0

index aa309aa..c79a983 100644
@@ -258,6 +258,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp)
 	if (atomic_dec_and_test(&fp->fi_delegees)) {
 		vfs_setlease(fp->fi_deleg_file, F_UNLCK, &fp->fi_lease);
 		fp->fi_lease = NULL;
+		fput(fp->fi_deleg_file);
 		fp->fi_deleg_file = NULL;
 	}
 }

Gerhard Heift

author

2011-04-20 11:08 UTC #6 lore raw

Quoted 15 lines · J. Bruce Fields

Quoted 14 lines · Gerhard Heift
Quoted 13 lines · J. Bruce Fields
Quoted 11 lines · Gerhard Heift
Quoted 7 lines · J. Bruce Fields
Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:
Should be fixed in more recent stable kernel; let me know if not.
An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.

Quoted 8 lines

Quoted 4 lines
Quoted 1 line
Ugh. OK, investigating.
It took me a bit of time, but I located the commit which introduced this
behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49
Doh--probably we just need this (lightly tested).
Thanks for tracking this down.

This patch works for me. Thanks for fixing it.

Quoted 1 line

--b.

Regards,
Gerhard

Quoted 24 lines

commit 21cd8aaf5efb086361cf12f9245c14a3b19fbafd
Author: J. Bruce Fields <[email protected]>
Date:   Fri Apr 15 18:08:26 2011 -0400
nfsd4: fix struct file leak on delegation
Introduced by acfdf5c383b38f7f4dddae41b97c97f1ae058f49.
Reported-by Gerhard Heift <[email protected]>
Signed-off-by J. Bruce Fields <[email protected]>
fs/nfsd/nfs4state.c +1 -0
index aa309aa..c79a983 100644
@@ -258,6 +258,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp)
 	if (atomic_dec_and_test(&fp->fi_delegees)) {
 		vfs_setlease(fp->fi_deleg_file, F_UNLCK, &fp->fi_lease);
 		fp->fi_lease = NULL;
+		fput(fp->fi_deleg_file);
 		fp->fi_deleg_file = NULL;
 	}
 }

J. Bruce Fields

2011-04-20 15:30 UTC #7 lore raw

Quoted 18 lines · Gerhard Heift

Quoted 17 lines · J. Bruce Fields
Quoted 16 lines · Gerhard Heift
Quoted 14 lines · J. Bruce Fields
Quoted 11 lines · Gerhard Heift
Quoted 7 lines · J. Bruce Fields
Quoted 4 lines · Gerhard Heift
Hello,
I have a linux kernel (2.6.38.2, armv5tel) running which exports a directory with following options:
Should be fixed in more recent stable kernel; let me know if not.
An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and there the leak happens too.
Ugh. OK, investigating.

Quoted 8 lines

Quoted 6 lines
Quoted 2 lines
It took me a bit of time, but I located the commit which introduced this
behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49
Doh--probably we just need this (lightly tested).
Thanks for tracking this down.
This patch works for me. Thanks for fixing it.

Thanks for the confirmation. I'll pass the patch along soon.

--b.