Skip to contentSkip to conversation

NFS caching problem

DISCUSSION David Warren 3 participants 4 messages Sep 26, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.

Program test runs on machine A writing to D:
(fortran)

     program test
     do i=1,10
        call system("/bin/rm t")
        open (10, file='t', status='new')
        write(10,*)i
        write(6,*)i
        close(10)
        call sleep(1)
     enddo
     end

program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;

while(1){
file=open("t", O_RDONLY);
len=read(file,in,79);
in[len]='\0';
printf("%s\n",in);
close(file);
}
}

while machine A is counting 1 - 10 and placing these numbers into file t, machine B is continually reading 1 from file t, then after a while it will switch to another number and read it for a while. In my first version of this, I was opening and rewriting the same file. In that version, machine B always read 1's. Now that I am creating new inodes all the time, it changes every few minutes while I repeatedly rerun test on machine A.

Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one would expect it to. As soon as the file is writen, the reader sees the new data.

Any ideas what I could have done wrong in my NFSv3 set up? Is there some kernel parameter that need tweaking? is there some mount option I should have??? Thanks.

Signature · 16 lines
David Warren 		INTERNET: [email protected]
(206) 543-0945		Fax: (206) 543-0308
University of Washington
Dept of Atmospheric Sciences, Box 351640
Seattle, WA 98195-1640




-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 36 lines · David Warren

I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.

Program test runs on machine A writing to D:
(fortran)

     program test
     do i=1,10
        call system("/bin/rm t")
        open (10, file='t', status='new')
        write(10,*)i
        write(6,*)i
        close(10)
        call sleep(1)
     enddo
     end

program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;

 while(1){
   file=open("t", O_RDONLY);
   len=read(file,in,79);
   in[len]='\0';
   printf("%s\n",in);
   close(file);

}
}

Quoted 7 lines

while machine A is counting 1 - 10 and placing these numbers into file t, machine B is continually reading 1 from file t, then after a while it will switch to another number and read it for a while. In my first version of this, I was opening and rewriting the same file. In that version, machine B always read 1's. Now that I am creating new inodes all the time, it changes every few minutes while I repeatedly rerun test on machine A.

Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one would expect it to. As soon as the file is writen, the reader sees the new data.

Any ideas what I could have done wrong in my NFSv3 set up? Is there some kernel parameter that need tweaking? is there some mount option I should have???

What version(s) of Linux are on machines, A, B, C? What is the local file system on machine, C, which is on disk, D?

Thanx...

ps

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
In reply to David Warren

m=C3=A5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:

Quoted 47 lines

I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.
=20
Program test runs on machine A writing to D:
(fortran)

      program test
      do i=3D1,10
         call system("/bin/rm t")
         open (10, file=3D't', status=3D'new')
         write(10,*)i
         write(6,*)i
         close(10)
         call sleep(1)
      enddo
      end

=20
program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;
=20

  while(1){
    file=3Dopen("t", O_RDONLY);
    len=3Dread(file,in,79);
    in[len]=3D'\0';
    printf("%s\n",in);
    close(file);

}
}
=20
while machine A is counting 1 - 10 and placing these numbers into file=20 t, machine B is continually reading 1 from file t, then after a while it=20 will switch to another number and read it for a while. In my first=20 version of this, I was opening and rewriting the same file. In that=20 version, machine B always read 1's. Now that I am creating new inodes=20 all the time, it changes every few minutes while I repeatedly rerun test=20 on machine A.
=20
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same=20 result as machine B.

Quoted 8 lines

The same thing under NFSv4 does not do this. It works exactly as one=20 would expect it to. As soon as the file is writen, the reader sees the=20 new data.
=20
Any ideas what I could have done wrong in my NFSv3 set up? Is there some=20 kernel parameter that need tweaking? is there some mount option I should=20 have???
Thanks.

When I see that problem on my test-rig, it appears to be due to the reuse of inode numbers by my server. IOW: the file created by the fortran program always ends up having the same inode number (check this using 'ls -i t').

In that case, the client is indeed expected to have problems recognising that the file has changed w.r.t. the cache.

Cheers,
Trond

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
Quoted 41 lines · Trond Myklebust

m=E5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:
=20

Quoted 38 lines

I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.

Program test runs on machine A writing to D:
(fortran)

     program test
     do i=3D1,10
        call system("/bin/rm t")
        open (10, file=3D't', status=3D'new')
        write(10,*)i
        write(6,*)i
        close(10)
        call sleep(1)
     enddo
     end

program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;

 while(1){
   file=3Dopen("t", O_RDONLY);
   len=3Dread(file,in,79);
   in[len]=3D'\0';
   printf("%s\n",in);
   close(file);

}
}

while machine A is counting 1 - 10 and placing these numbers into file=20 t, machine B is continually reading 1 from file t, then after a while i=

t=20

Quoted 4 lines
Quoted 4 lines

will switch to another number and read it for a while. In my first=20 version of this, I was opening and rewriting the same file. In that=20 version, machine B always read 1's. Now that I am creating new inodes=20 all the time, it changes every few minutes while I repeatedly rerun tes=

t=20

Quoted 10 lines
Quoted 10 lines

on machine A.

Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same=20 result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one=20 would expect it to. As soon as the file is writen, the reader sees the=20 new data.

Any ideas what I could have done wrong in my NFSv3 set up? Is there som=

e=20

Quoted 1 line
Quoted 1 line

kernel parameter that need tweaking? is there some mount option I shoul=

d=20

Quoted 5 lines
Quoted 4 lines

have???
Thanks.
=20

Quoted 8 lines

When I see that problem on my test-rig, it appears to be due to the reuse of inode numbers by my server. IOW: the file created by the fortran program always ends up having the same inode number (check this using 'ls -i t').

In that case, the client is indeed expected to have problems recognising that the file has changed w.r.t. the cache.

Actually, the server is responsible for changing the file handle when the inode is reused. There is a semantic called a "generation count" which i= s
used to differentiate between instances of files (re)using the same inode number. If the server does not do this, then it is broken and relays thi= s
breakage on to clients as demonstrated here.

ps

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs