NFS caching problem
Subsystems (2)
I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.
Program test runs on machine A writing to D:
(fortran)
program test
do i=1,10
call system("/bin/rm t")
open (10, file='t', status='new')
write(10,*)i
write(6,*)i
close(10)
call sleep(1)
enddo
endprogram t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;
while(1){
file=open("t", O_RDONLY);
len=read(file,in,79);
in[len]='\0';
printf("%s\n",in);
close(file);
}
}
while machine A is counting 1 - 10 and placing these numbers into file t, machine B is continually reading 1 from file t, then after a while it will switch to another number and read it for a while. In my first version of this, I was opening and rewriting the same file. In that version, machine B always read 1's. Now that I am creating new inodes all the time, it changes every few minutes while I repeatedly rerun test on machine A.
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one would expect it to. As soon as the file is writen, the reader sees the new data.
Any ideas what I could have done wrong in my NFSv3 set up? Is there some kernel parameter that need tweaking? is there some mount option I should have??? Thanks.
Signature · 16 lines
David Warren INTERNET: [email protected] (206) 543-0945 Fax: (206) 543-0308 University of Washington Dept of Atmospheric Sciences, Box 351640 Seattle, WA 98195-1640 ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 36 lines · David Warren
I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.Program test runs on machine A writing to D:
(fortran)program test do i=1,10 call system("/bin/rm t") open (10, file='t', status='new') write(10,*)i write(6,*)i close(10) call sleep(1) enddo endprogram t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;while(1){ file=open("t", O_RDONLY); len=read(file,in,79); in[len]='\0'; printf("%s\n",in); close(file);}
}
Quoted 7 lines
while machine A is counting 1 - 10 and placing these numbers into file t, machine B is continually reading 1 from file t, then after a while it will switch to another number and read it for a while. In my first version of this, I was opening and rewriting the same file. In that version, machine B always read 1's. Now that I am creating new inodes all the time, it changes every few minutes while I repeatedly rerun test on machine A.
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one would expect it to. As soon as the file is writen, the reader sees the new data.Any ideas what I could have done wrong in my NFSv3 set up? Is there some kernel parameter that need tweaking? is there some mount option I should have???
What version(s) of Linux are on machines, A, B, C? What is the local file system on machine, C, which is on disk, D?
Thanx...
ps
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
m=C3=A5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:
Quoted 47 lines
I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.
=20
Program test runs on machine A writing to D:
(fortran)program test do i=3D1,10 call system("/bin/rm t") open (10, file=3D't', status=3D'new') write(10,*)i write(6,*)i close(10) call sleep(1) enddo end=20
program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;
=20while(1){ file=3Dopen("t", O_RDONLY); len=3Dread(file,in,79); in[len]=3D'\0'; printf("%s\n",in); close(file);}
}
=20
while machine A is counting 1 - 10 and placing these numbers into file=20 t, machine B is continually reading 1 from file t, then after a while it=20 will switch to another number and read it for a while. In my first=20 version of this, I was opening and rewriting the same file. In that=20 version, machine B always read 1's. Now that I am creating new inodes=20 all the time, it changes every few minutes while I repeatedly rerun test=20 on machine A.
=20
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same=20 result as machine B.
Quoted 8 lines
The same thing under NFSv4 does not do this. It works exactly as one=20 would expect it to. As soon as the file is writen, the reader sees the=20 new data.
=20
Any ideas what I could have done wrong in my NFSv3 set up? Is there some=20 kernel parameter that need tweaking? is there some mount option I should=20 have???
Thanks.
When I see that problem on my test-rig, it appears to be due to the reuse of inode numbers by my server. IOW: the file created by the fortran program always ends up having the same inode number (check this using 'ls -i t').
In that case, the client is indeed expected to have problems recognising that the file has changed w.r.t. the cache.
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 41 lines · Trond Myklebust
m=E5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:
=20Quoted 38 lines
I have discovered a wierd problem with NFSv3 on linux. I have 3 machines
machine A and B both mount a disk D from machine C The options are tcp,rw,hard and intr.Program test runs on machine A writing to D:
(fortran)program test do i=3D1,10 call system("/bin/rm t") open (10, file=3D't', status=3D'new') write(10,*)i write(6,*)i close(10) call sleep(1) enddo endprogram t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;while(1){ file=3Dopen("t", O_RDONLY); len=3Dread(file,in,79); in[len]=3D'\0'; printf("%s\n",in); close(file);}
}while machine A is counting 1 - 10 and placing these numbers into file=20 t, machine B is continually reading 1 from file t, then after a while i=
t=20
Quoted 4 lines
Quoted 4 lines
will switch to another number and read it for a while. In my first=20 version of this, I was opening and rewriting the same file. In that=20 version, machine B always read 1's. Now that I am creating new inodes=20 all the time, it changes every few minutes while I repeatedly rerun tes=
t=20
Quoted 10 lines
Quoted 10 lines
on machine A.
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same=20 result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one=20 would expect it to. As soon as the file is writen, the reader sees the=20 new data.Any ideas what I could have done wrong in my NFSv3 set up? Is there som=
e=20
Quoted 1 line
Quoted 1 line
kernel parameter that need tweaking? is there some mount option I shoul=
d=20
Quoted 5 lines
Quoted 4 lines
have???
Thanks.
=20
Quoted 8 lines
When I see that problem on my test-rig, it appears to be due to the reuse of inode numbers by my server. IOW: the file created by the fortran program always ends up having the same inode number (check this using 'ls -i t').
In that case, the client is indeed expected to have problems recognising that the file has changed w.r.t. the cache.
Actually, the server is responsible for changing the file handle when the inode is reused. There is a semantic called a "generation count" which i= s
used to differentiate between instances of files (re)using the same inode number. If the server does not do this, then it is broken and relays thi= s
breakage on to clients as demonstrated here.
ps
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs