file open for write fails under stress
Subsystems (2)
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
my application is single threaded, and I noticed the following problem when running a single instance of it:
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
if I sleep for 1 second and try again, it works.
my NFS server runs Debian etch 32 bit, with
nfs-kernel-server and nfs-common (1:1.0.10-6+etch.1).
client is also etch 32.
kernel on both is :2.6.18-6-686 #1 SMP Tue Jun 17 21:31:27 UTC 2008 i686
both machines are quad core.
I tried to mount with tcp and udp, and tried both sync and async export point.
is this a known issue?
any known workaround that will not require messing up the application with retry logic?
Omry.
Quoted 4 lines · Omry Yadan
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
Quoted 8 lines
my application is single threaded, and I noticed the following problem when running a single instance of it:
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
if I sleep for 1 second and try again, it works.
What version of nfs, filesystem on the server, export options, mount options,...?
Cheers
Trond
Quoted 10 lines · Trond Myklebust
Quoted 8 lines · Omry Yadan
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
my application is single threaded, and I noticed the following problem when running a single instance of it:
Quoted 10 lines
Quoted 5 lines
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
if I sleep for 1 second and try again, it works.
What version of nfs, filesystem on the server, export options, mount options,...?
/etc/exports:/var/vizi 38.99.126.160/28(rw,sync,no_subtree_check) server file system : ext3 on a raid5 (900gb, 3x500gb hd). server nfs version: 1.0.10
client mount:
nas1:/var/vizi on /var/vizi type nfs (rw,tcp,addr=38.99.126.163)
Quoted 23 lines · Omry Yadan
Quoted 10 lines · Trond Myklebust
Quoted 8 lines · Omry Yadan
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
my application is single threaded, and I noticed the following problem when running a single instance of it:
Quoted 10 lines
Quoted 5 lines
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
if I sleep for 1 second and try again, it works.
What version of nfs, filesystem on the server, export options, mount options,...?
Quoted 7 lines
/etc/exports:/var/vizi38.99.126.160/28(rw,sync,no_subtree_check) server file system : ext3 on a raid5 (900gb, 3x500gb hd). server nfs version: 1.0.10client mount:
nas1:/var/vizi on/var/vizitype nfs (rw,tcp,addr=38.99.126.163)
So I assume it is NFS version 3, then?
What error does the application get when the open() fails?
Cheers
Trond
Quoted 8 lines
Quoted 6 lines
/etc/exports:/var/vizi38.99.126.160/28(rw,sync,no_subtree_check) server file system : ext3 on a raid5 (900gb, 3x500gb hd). server nfs version: 1.0.10
Quoted 8 lines
Quoted 3 lines
client mount:
nas1:/var/vizi on/var/vizitype nfs (rw,tcp,addr=38.99.126.163)So I assume it is NFS version 3, then?
What error does the application get when the
open()fails?
according to nfsstat, yes - it's NFS v3.
after open fails (I use fstream binary_file(name,ios::out|ios::binary) ) the error message returned from strerror_r(errno, buf, 1000) is something like "Input Output error".
Quoted 10 lines · Omry Yadan
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
my application is single threaded, and I noticed the following problem when running a single instance of it:
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
Do you know what error exactly the open is failing with?
--b.
Quoted 33 lines
if I sleep for 1 second and try again, it works.
my NFS server runs Debian etch 32 bit, with
nfs-kernel-server and nfs-common (1:1.0.10-6+etch.1).
client is also etch 32.
kernel on both is :2.6.18-6-686 #1 SMP Tue Jun 17 21:31:27 UTC 2008 i686
both machines are quad core.
I tried to mount with tcp and udp, and tried both sync and async export point.
is this a known issue?
any known workaround that will not require messing up the application with retry logic?
Omry.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · J. Bruce Fields
Quoted 8 lines · Omry Yadan
I have an application that open many files for writing very fast, in a way - it can almost be looked as a file system stress test.
my application is single threaded, and I noticed the following problem when running a single instance of it:
Quoted 6 lines
Quoted 3 lines
at some point, after opening (and closing) many files for writing, the application fails to open a file for writing.
Do you know what error exactly the open is failing with?
As I wrote in a separate message to the list:
the error string is "Input Output error".
I can get the exact errno if you like, but it's probably a generic IO error code.