Data corruption using 2.6.18 NFSv3 client -- sparse files?
Subsystems (2)
Hello,
When multiple 2.6.18 NFSv3 clients write to the same file, after one of the clients has recently read from the file, we see data corruption in the form of null bytes inserted into the file.
Simple test case:
hosta% echo "line 1" > /nfs/volume/bar.txt
then, in rapid succession:
hostb% cat bar.txt && sleep 2 && echo "line 2 from hostb" >> /nfs/bar.txt hostc% cat bar.txt && sleep 2 && echo "line 2 from hostc" >> /nfs/bar.txt
Expected result:
/nfs/bar.txt contains:
line 1
line 2 from hostb
line 2 from hostc
Actual result:
/nfs/bar.txt contains:
line 1 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0line 2 from hostc
This seems to be due to an inconsistency between the page cache and the attribute cache on hostc. Running the 'cat' command on hostc causes bar.txt to be loaded into the page cache. Meanwhile, its attributes are cached in the attribute cache.
Seconds later, the 'echo' command on hostc causes the attribute cache to be updated (a GETATTR operation is issued) with the new file size (reflecting the line just appended by hostb), but the page cache is not updated (no READ operation is issued).
The subsequent WRITE operation from hostc specifies an offset of 0 (beginning of file) and a length equal to "line 1" + "line 2 from hostb" + "line 2 from hostc". Since the page cache on hostc does not contain the "line 2 from hostb" content, that segment of the WRITE buffer is filled with nulls.
Note that no file locking is being used in this test case or our production use case.
Questions:
- Is this a bug or correct operation?
- Would file locking produce the expected behaviour?
Thanks,
Clay McClure
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 48 lines · Clay McClure
Hello,
When multiple 2.6.18 NFSv3 clients write to the same file, after one of the clients has recently read from the file, we see data corruption in the form of null bytes inserted into the file.
Simple test case:
hosta% echo "line 1" > /nfs/volume/bar.txt
then, in rapid succession:
hostb% cat bar.txt && sleep 2 && echo "line 2 from hostb" >> /nfs/bar.txt hostc% cat bar.txt && sleep 2 && echo "line 2 from hostc" >> /nfs/bar.txt
Expected result:
/nfs/bar.txt contains:
line 1
line 2 from hostb
line 2 from hostcActual result:
/nfs/bar.txt contains:
line 1 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0line 2 from hostc
This seems to be due to an inconsistency between the page cache and the attribute cache on hostc. Running the 'cat' command on hostc causes bar.txt to be loaded into the page cache. Meanwhile, its attributes are cached in the attribute cache.
Seconds later, the 'echo' command on hostc causes the attribute cache to be updated (a GETATTR operation is issued) with the new file size (reflecting the line just appended by hostb), but the page cache is not updated (no READ operation is issued).
The subsequent WRITE operation from hostc specifies an offset of 0 (beginning of file) and a length equal to "line 1" + "line 2 from hostb" + "line 2 from hostc". Since the page cache on hostc does not contain the "line 2 from hostb" content, that segment of the WRITE buffer is filled with nulls.
Note that no file locking is being used in this test case or our production use case.
Quoted 8 lines
Questions:
- Is this a bug or correct operation?
- Would file locking produce the expected behaviour?Thanks,
Clay McClure
A number of read and write races have been fixed since September 2006. Have you tested with 2.6.25?
Trond
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Clay" == Clay McClure <clay-zo/yENU/[email protected]> writes:
Clay> Hello, When multiple 2.6.18 NFSv3 clients write to the same Clay> file, after one of the clients has recently read from the Clay> file, we see data corruption in the form of null bytes Clay> inserted into the file.
If its the same bug we've seen it appeared in 2.6.17 and we "fixed" by downgrading the involved hosts to 2.6.16. It was related to this post by Trond Myklebust (in Jan/Feb) :-
That is very likely to be the same problem as fixed in
AFAICR, all kernels after 2.6.17 were affected. Can you therefore see if the same patch applies to 2.6.19, and if it fixes your problem?
Cheers
Trond
Sincerely,
Adrian Phillips
Signature · 15 lines
Who really wrote the works of William Shakespeare ? http://www.pbs.org/wgbh/pages/frontline/shakespeare/ ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs