NFS cache problem
Subsystems (2)
I have fileserver exporting via NFSv3 /home directories to few desktops and small cluster. File server has two NICs, one for cluster, one for desktops. Kernel version is 2.6.5.
export options are (exportfs -v):
/home 192.168.211.0/24(rw,async,no_root_squash)
mount options are (cat /proc/mounts):
192.168.211.240:/home /home nfs rw,sync,v3,rsize=32768,wsize=32768,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,tcp,noac,lock,addr=192.168.211.240 0 0
Time on all clients (desktops and cluster) are synchronized via NTP.
But time to time I have strange situation. You can rewrite file at one host, but for long time (up to few hours!!!) some hosts will read new file, some hosts will read old file.
I tried to play with all options without any result. Exporting with "sync" seems to be a sollution, but performance is really very low (because of this "sync" wasn't realy tested). And and I don't need "sync" because I'm not interesting in saving to "real" media. File-cache is good enough for me. And, in principal, "syns/async" on server side should be irrelevant in this case (at least on client side I have "sync", it's enough).
To avoid future discussion, there is nothing "cluster specific". Cluster is very small and there is nothing like concurent read/write. Basically, only one specific thing, the same data can be accesible from different host. But even not at the same time usually.
You write file on one host, let say, and in couple minutes you read it from different host (you prepare input data on client host or master node and after you submit job into the queue). That should be OK, but in my case...up to few hours clients can see chaos by reading different versions of file.
Does anybody has some ideas how to solve the problem?
BTW, hardware configuration.
server:
3ware SATA raid, 2xXeon CPUs (NFSD started in 8 threads). Intel and Broadcom GbE NICs.
Clients - mostly dual Opteron machines.
Actually, I have strong filling that problem started to be much more "visible" when I have added second CPU to server. It exists before, but usually not longer that for 10 minutes. Now my users report me about hours. This is incredible. Basically, work in my group partly paralysed now :(
Of course there is such things like lustre, PVFS and so on. But I beleive that my case isn't proper case to start use such filesystems. NFS should be more than enough.
Thanks,
Anton Starikov.
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
to den 26.05.2005 Klokka 01:01 (+0200) skreiv Anton Starikov:
Quoted 37 lines
I have fileserver exporting via NFSv3 /home directories to few desktops and small cluster. File server has two NICs, one for cluster, one for desktops. Kernel version is 2.6.5.
export options are (exportfs -v):
/home 192.168.211.0/24(rw,async,no_root_squash)mount options are (cat
/proc/mounts):
192.168.211.240:/home /home nfs rw,sync,v3,rsize=32768,wsize=32768,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,tcp,noac,lock,addr=192.168.211.240 0 0Time on all clients (desktops and cluster) are synchronized via NTP.
But time to time I have strange situation. You can rewrite file at one host, but for long time (up to few hours!!!) some hosts will read new file, some hosts will read old file.
I tried to play with all options without any result. Exporting with "sync" seems to be a sollution, but performance is really very low (because of this "sync" wasn't realy tested). And and I don't need "sync" because I'm not interesting in saving to "real" media. File-cache is good enough for me. And, in principal, "syns/async" on server side should be irrelevant in this case (at least on client side I have "sync", it's enough).
To avoid future discussion, there is nothing "cluster specific". Cluster is very small and there is nothing like concurent read/write. Basically, only one specific thing, the same data can be accesible from different host. But even not at the same time usually.
You write file on one host, let say, and in couple minutes you read it from different host (you prepare input data on client host or master node and after you submit job into the queue). That should be OK, but in my case...up to few hours clients can see chaos by reading different versions of file.
Quoted 1 line
Does anybody has some ideas how to solve the problem?
The NFS cache consistency model is discussed in the FAQ. Please see
http://nfs.sourceforge.net/#faq_a8
Cheers,
Trond
Quoted 22 lines
BTW, hardware configuration.
server:
3ware SATA raid, 2xXeon CPUs (NFSD started in 8 threads). Intel and Broadcom GbE NICs.Clients - mostly dual Opteron machines.
Actually, I have strong filling that problem started to be much more "visible" when I have added second CPU to server. It exists before, but usually not longer that for 10 minutes. Now my users report me about hours. This is incredible. Basically, work in my group partly paralysed now :(
Of course there is such things like lustre, PVFS and so on. But I beleive that my case isn't proper case to start use such filesystems. NFS should be more than enough.
Thanks,
Anton Starikov.
Quoted 8 lines
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Trond Myklebust
The NFS cache consistency model is discussed in the FAQ. Please see
http://nfs.sourceforge.net/#faq_a8
I did it long time ago. I've reread it now.
My usage fits this. I have "noac" and "sync" on client side. What's more? :)
Can ACL be a problem?
Best regards,
Anton
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
to den 26.05.2005 Klokka 01:44 (+0200) skreiv Anton Starikov:
Quoted 6 lines
Quoted 2 lines · Trond Myklebust
The NFS cache consistency model is discussed in the FAQ. Please see
http://nfs.sourceforge.net/#faq_a8I did it long time ago. I've reread it now.
My usage fits this. I have "noac" and "sync" on client side. What's more? :)
"noac" just means that the client rereads the attributes every time it reads or writes.
If the size and the mtime on the file haven't changed, then the client assumes the data in the file itself also hasn't changed. That is usually a good assumption on something like XFS (which has nanosecond precision on mtime). It can be rather inaccurate on something like EXT3 (which has a precision of only 1 second on mtime).
If you seriously need uncached reads and writes, then you should rather consider using O_DIRECT.
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Trond Myklebust
"noac" just means that the client rereads the attributes every time it reads or writes.
If the size and the mtime on the file haven't changed, then the client assumes the data in the file itself also hasn't changed. That is usually a good assumption on something like XFS (which has nanosecond precision on mtime). It can be rather inaccurate on something like EXT3 (which has a precision of only 1 second on mtime).
I use ReiserFS.
And here we talk not about seconds...we talk about hours sometimes. That's seems too strange for me. For seconds I can find plenty of explanations :)
In principal even if it will be one minute, I'll be much more happy than now.
Quoted 2 lines
If you seriously need uncached reads and writes, then you should rather, consider using
O_DIRECT.
Unfortunatelly this is not trivial problem. A lot of software are involved here. And I beleive that NFS should be able to work in this conditions properly. At least it did with solaris in similar environment.
Best,
Anton.
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
to den 26.05.2005 Klokka 02:29 (+0200) skreiv Anton Starikov:
Quoted 8 lines
Quoted 7 lines · Trond Myklebust
"noac" just means that the client rereads the attributes every time it reads or writes.
If the size and the mtime on the file haven't changed, then the client assumes the data in the file itself also hasn't changed. That is usually a good assumption on something like XFS (which has nanosecond precision on mtime). It can be rather inaccurate on something like EXT3 (which has a precision of only 1 second on mtime).
Quoted 6 lines
I use ReiserFS.
And here we talk not about seconds...we talk about hours sometimes. That's seems too strange for me. For seconds I can find plenty of explanations :)
In principal even if it will be one minute, I'll be much more happy than now.
I repeat: if the file attributes do not change, then the NFS client will not update its data cache. That is true of Solaris too.
Quoted 5 lines
Quoted 2 lines
If you seriously need uncached reads and writes, then you should rather, consider using
O_DIRECT.Unfortunatelly this is not trivial problem. A lot of software are involved here. And I beleive that NFS should be able to work in this conditions properly. At least it did with solaris in similar environment.
Mind describing _what_ kind of environment it works under? In other words, what is the Solaris client doing that the Linux client is not doing when running under the _same_ conditions (i.e. against the same server and running the same applications)?
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs