Data coherency trouble with multiple clients, on 2.6.14-rc5
Subsystems (2)
Hi all,
I'm hoping someone can shed some light on this problem I'm having.
I have 2 NFS client machines (CentOS 4, kernel 2.6.14-rc5+latest 'NFS_ALL'=
patch, nfsutils 1.0.6) who each mount the same export from an NFS server (=
WBEL3, kernel 2.4.21-20, nfsutils 1.0.6).
These client machines are supposed to run an in-house database application=
- multiple processes on different clients often have the same datafile op=
en simultaneously (via NFS).
Previously, we had these client machines running 2.2.21 kernels. The old (= pre-CTO) behaviour would see writes from any of the clients becoming immed= iately 'visible' in any of the others. This is no longer the case, even wi= th 'nocto' specified as a mount option.
Currently, if a process on client1 has the file open while it is written t=
o by another process (on client2), the changes made to the file are never r= eflected in the data available to the process on client1 unless:
a) client1's NFS filesystem is unmounted and re-mounted
b) The file is 'touched' on the server or on client2 after both processes h= ave closed the file
c) The process on client1 writes to the file (note 'touching' the file on c= lient1 has no effect, either while or after the process has the file open)= .
It doesn't matter if both processes subsequently close the file and exit, t= hen re-open the file from scratch - the discrepancy remains until either (= a), (b) or (c) occurs. Client1 and client2 are identical systems, and the p= roblem works both ways. In the event of (c), the previous write becomes vi= sible to client1, but the write by client1 is not visible to client2.
'stat' run on the datafile from either client always gives the same output= .
I've also tried adding O_SYNC to the flags in the call to 'open()' in the a= pplication itself. Makes no difference.=C2=A0 fsync()ing after every write=
similarly makes no difference to the problem.
This problem appears similar to that which the linux-2.6.14-01-fix_attrcac= he patch was posted in response to, but it can't be reproduced with the pr= ocedure outlined in that post.
The client mount options are: noac,nocto,sync,hard,intr,rw,nfsvers=3D3 The server exports flags are: (rw,no_root_squash,sync,no_wdelay,insecure)
Various combinations of the above mount options make no difference. I've also tried with a 2.6.14-rc5 server, and the problem persists.
Thanks,
---
Charles Duffy
Systems Administrator
Mail Call Couriers
-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
on den 26.10.2005 klokka 16:01 (+1000) skreiv Charles Duffy:
Quoted 9 lines
Hi all,
I'm hoping someone can shed some light on this problem I'm having.
I have 2 NFS client machines (CentOS 4, kernel 2.6.14-rc5+latest '
NFS_ALL' patch, nfsutils 1.0.6) who each mount the same export from an NFS server (WBEL3, kernel 2.4.21-20, nfsutils 1.0.6).These client machines are supposed to run an in-house database application - multiple processes on different clients often have the same datafile open simultaneously (via NFS).
Previously, we had these client machines running 2.2.21 kernels. The old (pre-CTO) behaviour would see writes from any of the clients becoming immediately 'visible' in any of the others. This is no longer the case, even with 'nocto' specified as a mount option.
"nocto" gives you stronger caching than close-to-open. It is designed for the case where a single client has sole access to the files.
Quoted 12 lines
Currently, if a process on client1 has the file open while it is written to by another process (on client2), the changes made to the file are never reflected in the data available to the process on client1 unless:
a) client1's NFS filesystem is unmounted and re-mounted
b) The file is 'touched' on the server or on client2 after both processes have closed the file
c) The process on client1 writes to the file (note 'touching' the file on client1 has no effect, either while or after the process has the file open).It doesn't matter if both processes subsequently close the file and exit, then re-open the file from scratch - the discrepancy remains until either (a), (b) or (c) occurs. Client1 and client2 are identical systems, and the problem works both ways. In the event of (c), the previous write becomes visible to client1, but the write by client1 is not visible to client2.
'stat' run on the datafile from either client always gives the same output.
I've also tried adding
O_SYNCto the flags in the call to 'open()' in the application itself. Makes no difference. fsync()ing after every write similarly makes no difference to the problem.
Quoted 7 lines
This problem appears similar to that which the linux-2.6.14-01-fix_attrcache patch was posted in response to, but it can't be reproduced with the procedure outlined in that post.
The client mount options are: noac,nocto,sync,hard,intr,rw,nfsvers=3 The server exports flags are: (rw,no_root_squash,sync,no_wdelay,insecure)
Various combinations of the above mount options make no difference. I've also tried with a 2.6.14-rc5 server, and the problem persists.
It sounds like you are really looking for the O_DIRECT (i.e. uncached) file I/O mode.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: Data coherency trouble with multiple clients, on2.6.14-rc5
Trond,
Thanks alot...O_DIRECT does the trick (at least in initial testing). Removing the 'nocto' mount flag made no difference, though.
Quoted 31 lines
on den 26.10.2005 klokka 16:01 (+1000) skreiv Charles Duffy:
Quoted 9 lines
Hi all,
I'm hoping someone can shed some light on this problem I'm having.
I have 2 NFS client machines (CentOS 4, kernel 2.6.14-rc5+latest '
NFS_ALL' patch, nfsutils 1.0.6) who each mount the same export from an NFS server (WBEL3, kernel 2.4.21-20, nfsutils 1.0.6).These client machines are supposed to run an in-house database application - multiple processes on different clients often have the same datafile open simultaneously (via NFS).
Previously, we had these client machines running 2.2.21 kernels. The old (pre-CTO) behaviour would see writes from any of the clients becoming immediately 'visible' in any of the others. This is no longer the case, even with 'nocto' specified as a mount option.
"nocto" gives you stronger caching than close-to-open. It is designed for the case where a single client has sole access to the files.
Quoted 17 lines
Currently, if a process on client1 has the file open while it is written to by another process (on client2), the changes made to the file are never reflected in the data available to the process on client1 unless:
a) client1's NFS filesystem is unmounted and re-mounted
b) The file is 'touched' on the server or on client2 after both processes have closed the file
c) The process on client1 writes to the file (note 'touching' the file on client1 has no effect, either while or after the process has the file open).It doesn't matter if both processes subsequently close the file and exit, then re-open the file from scratch - the discrepancy remains until either (a), (b) or (c) occurs. Client1 and client2 are identical systems, and the problem works both ways. In the event of (c), the previous write becomes visible to client1, but the write by client1 is not visible to client2.
'stat' run on the datafile from either client always gives the same output.
I've also tried adding
O_SYNCto the flags in the call to 'open()' in the application itself. Makes no difference. fsync()ing after every write similarly makes no difference to the problem.This problem appears similar to that which the linux-2.6.14-01-fix_attrcache patch was posted in response to, but it can't be reproduced with the procedure outlined in that post.
The client mount options are: noac,nocto,sync,hard,intr,rw,nfsvers=3 The server exports flags are: (rw,no_root_squash,sync,no_wdelay,insecure)
Quoted 9 lines
Quoted 2 lines
Various combinations of the above mount options make no difference. I've also tried with a 2.6.14-rc5 server, and the problem persists.
It sounds like you are really looking for the
O_DIRECT(i.e. uncached) file I/O mode.Cheers,
Trond
---
Charles Duffy
Systems Administrator
Mail Call Couriers
-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs