Skip to contentSkip to conversation

New file doesn't show up if cached as missing

DISCUSSION Simon Kirby 2 participants 4 messages May 29, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello!

On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can fairly easily reproduce a case where creating a file (open,write,close) and then attempting to open it on another client mounting the same server returns ENOENT. If I run "ls" before I try to cat it, it works.

To reproduce:

ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
ssh root@lps02 'echo hi > /shared/ssl/test'; \
ssh root@lps01 'cat /shared/ssl/test'

The original stat (or cat or open) is important as it seems to cache that the file does not exist. I can then even log in and poke around:

lps01:~# cd /shared
lps01:/shared# cat test
cat: test: No such file or directory
lps01:/shared# stat test
stat: cannot stat `test': No such file or directory
lps01:/shared# ls test
ls: test: No such file or directory
lps01:/shared# ls
<files including "test">
lps01lsh:/shared# cat test
hi

Is this expected behaviour? The file system is mounted on both clients
with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac

(eg: we can't find a way to make it not happen.)

Cheers,

Simon-

#1 lore raw
Quoted 28 lines · Simon Kirby

Hello!

On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can fairly easily reproduce a case where creating a file (open,write,close) and then attempting to open it on another client mounting the same server returns ENOENT. If I run "ls" before I try to cat it, it works.

To reproduce:

ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
ssh root@lps02 'echo hi > /shared/ssl/test'; \
ssh root@lps01 'cat /shared/ssl/test'

The original stat (or cat or open) is important as it seems to cache that the file does not exist. I can then even log in and poke around:

lps01:~# cd /shared
lps01:/shared# cat test
cat: test: No such file or directory
lps01:/shared# stat test
stat: cannot stat `test': No such file or directory
lps01:/shared# ls test
ls: test: No such file or directory
lps01:/shared# ls
<files including "test">
lps01lsh:/shared# cat test
hi
Quoted 4 lines

Is this expected behaviour? The file system is mounted on both clients
with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac

(eg: we can't find a way to make it not happen.)

That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.

Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).

Trond

#2 lore raw
Quoted 4 lines · Trond Myklebust

That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.

As in "ls --full-time" shows 0 for the fractional second part?

Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...

Quoted 2 lines

Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).

After backporting nfs-utils, either of those options seem to this particular case work as desired. Excellent!

I guess all of the attribute-caching-related options are unrelated to this case. Is this consistent with older operation and that of other OSes? It might be a little surprising for some (me) when "noac" still ends up doing some caching.

Thanks,

Simon-

#3 lore raw
Quoted 10 lines · Simon Kirby
Quoted 4 lines · Trond Myklebust

That is most likely to be a consequence of poor mtime resolution on the server (i.e. the directory mtime failing to change because the file creation occurred within < 1 second of the 'rm'), combined with negative lookup caching.

As in "ls --full-time" shows 0 for the fractional second part?

Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...

No. You'll have to migrate to ext4 (or xfs) if you want finer grained timestamps.

Quoted 10 lines
Quoted 2 lines

Try using the '-olookupcache=positive' or '-olookupcache=none' mount options (requires a relatively recent version of nfs-utils).

After backporting nfs-utils, either of those options seem to this particular case work as desired. Excellent!

I guess all of the attribute-caching-related options are unrelated to this case. Is this consistent with older operation and that of other OSes? It might be a little surprising for some (me) when "noac" still ends up doing some caching.

noac is all about attribute caching, whereas this is lookup/dentry caching. You can compare it to data caching: 'noac' doesn't force the cache to be flushed on every read() request, but it does force the mtime to be checked, and so cache consistency is relatively stronger (provided the mtime is accurate enough).

Cheers
Trond