New file doesn't show up if cached as missing
Subsystems (2)
Hello!
On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can fairly easily reproduce a case where creating a file (open,write,close) and then attempting to open it on another client mounting the same server returns ENOENT. If I run "ls" before I try to cat it, it works.
To reproduce:
ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
ssh root@lps02 'echo hi > /shared/ssl/test'; \
ssh root@lps01 'cat /shared/ssl/test'
The original stat (or cat or open) is important as it seems to cache that the file does not exist. I can then even log in and poke around:
lps01:~# cd /shared lps01:/shared# cat test cat: test: No such file or directory lps01:/shared# stat test stat: cannot stat `test': No such file or directory lps01:/shared# ls test ls: test: No such file or directory lps01:/shared# ls <files including "test"> lps01lsh:/shared# cat test hi
Is this expected behaviour? The file system is mounted on both clients
with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac
(eg: we can't find a way to make it not happen.)
Cheers,
Simon-
Quoted 28 lines · Simon Kirby
Hello!
On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can fairly easily reproduce a case where creating a file (open,write,close) and then attempting to open it on another client mounting the same server returns ENOENT. If I run "ls" before I try to cat it, it works.
To reproduce:
ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
ssh root@lps02 'echo hi > /shared/ssl/test'; \
ssh root@lps01 'cat /shared/ssl/test'The original stat (or cat or open) is important as it seems to cache that the file does not exist. I can then even log in and poke around:
lps01:~# cd /shared lps01:/shared# cat test cat: test: No such file or directory lps01:/shared# stat test stat: cannot stat `test': No such file or directory lps01:/shared# ls test ls: test: No such file or directory lps01:/shared# ls <files including "test"> lps01lsh:/shared# cat test hi
Quoted 4 lines
Is this expected behaviour? The file system is mounted on both clients
with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac(eg: we can't find a way to make it not happen.)
That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.
Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).
Trond
Quoted 4 lines · Trond Myklebust
That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.
As in "ls --full-time" shows 0 for the fractional second part?
Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...
Quoted 2 lines
Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).
After backporting nfs-utils, either of those options seem to this particular case work as desired. Excellent!
I guess all of the attribute-caching-related options are unrelated to this case. Is this consistent with older operation and that of other OSes? It might be a little surprising for some (me) when "noac" still ends up doing some caching.
Thanks,
Simon-
Quoted 10 lines · Simon Kirby
Quoted 4 lines · Trond Myklebust
That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.
As in "ls --full-time" shows 0 for the fractional second part?
Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...
No. You'll have to migrate to ext4 (or xfs) if you want finer grained timestamps.
Quoted 10 lines
Quoted 2 lines
Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).
After backporting nfs-utils, either of those options seem to this particular case work as desired. Excellent!
I guess all of the attribute-caching-related options are unrelated to this case. Is this consistent with older operation and that of other OSes? It might be a little surprising for some (me) when "noac" still ends up doing some caching.
noac is all about attribute caching, whereas this is lookup/dentry caching. You can compare it to data caching: 'noac' doesn't force the cache to be flushed on every read() request, but it does force the mtime to be checked, and so cache consistency is relatively stronger (provided the mtime is accurate enough).
Cheers
Trond