Problem related to directory caches
Subsystems (2)
Hello, there.
First of all, I'd like to thank all on the list for helping with my previous doubts, it was very helpful.
Now, I'm having a problem, and I believe it may be related to cache of directory entries. My environment is:
- clients are Linux RH7.3 with kernel 2.4.9
- servers are EMC Celerra, NS700 and Netapp FAS740
- mount options are rw,vers=3,proto=udp,nolock,rsize=8192,wsize=8192
The behaviour I'm getting is: I'm creating files in a directory, only creating not removing, but sometimes, one of the clients reports a number of files that is less than one other client reported some time ago. For example, this is what I have in logs:
May 1 05:49:00 cabue trrmfsbuilder[23479]: ... Msgs [20], Read [0], New [20], Bytes [179742]
May 1 05:49:02 lusaca trrmfsbuilder[11519]: ... Msgs [19], Read [0], New [19], Bytes [176448]
What these logs mean:
1) At 5:49:00, host cabue found 20 files on the directory
2) At 5:49:02 (two seconds later), host lusaca found only 19 files on the same directory.
This is a NFS directory. I checked time differences, but both hosts are running NTP (without problems) and they certainly don't have as much as 2 seconds difference, they certainly have less than 1 second difference...
Can this be caused by NFS cache of the directory entry? I know that host lusaca probably saw this same directory some seconds before it counted 19 files, so that I believe this could be cached somehow and causing the problem of not finding the 20 files that probably were there...
If the problem is this, how can I solve it?
I thought of two solutions, but I have some questions about it:
1) I could set flags such as "noac" or "acdirmax=0" or something like that (what would be the right ones to force no cache for directory entries?). But I also would like to know what other impact can I have for doing this, I guess that probably changing this would impact other applications as well...
2) I read about using "fcntl" with "FLOCK" to flush a file's cache on Linux (even mounting with "nolock" this is supposed to work). Is that so? Can I use it on a directory entry? In that case, should I fcntl the directory itself?
Well, what do you think? Can you help me on this? Can you tell me if on of my two options would give me data that is not subject to inconsistency because of the cache? Can you tell me if there is a better way to do it? Can a more recent kernel solve this kind of problem?
Thanks a lot on advance! I hope you can help!
Filipe
-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
to den 05.05.2005 Klokka 18:08 (-0300) skreiv Filipe Brandenburger:
Quoted 7 lines
I thought of two solutions, but I have some questions about it:
1) I could set flags such as "noac" or "acdirmax=0" or something like that (what would be the right ones to force no cache for directory entries?). But I also would like to know what other impact can I have for doing this, I guess that probably changing this would impact other applications as well...
It would mainly impact the server since it will be bombarded with GETATTR calls every time you do something in a directory.
Quoted 4 lines
2) I read about using "fcntl" with "FLOCK" to flush a file's cache on Linux (even mounting with "nolock" this is supposed to work). Is that so? Can I use it on a directory entry? In that case, should I fcntl the directory itself?
You would have to set a read lock, since you cannot open a directory for writing. That would normally suffice to flush the readdir cache, though.
Quoted 4 lines
Well, what do you think? Can you help me on this? Can you tell me if on of my two options would give me data that is not subject to inconsistency because of the cache? Can you tell me if there is a better way to do it? Can a more recent kernel solve this kind of problem?
No. This will not suffice to ensure consistency.
Even on a single client, 2 threads that both call readdir() and then print out the result can race. The thread that called readdir() first can get preemptively scheduled before it prints the result, the printing may be buffered (as I believe is the case for syslogd), etc... The only way to ensure that they don't race is to use a mutex or some other form of lock to ensure that the threads synchronise the operation readdir()+printf()+flush().
The same sort of thing applies when you are comparing the output from two different processes running on different NFS clients: you would need some form of exclusive lock in order to synchronise the two processes. Most of the time, it is not worth the trouble.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs