blocks of zeros (NULLs) in NFS files in kernels >= 2.6.20
Subsystems (2)
Hi all,
We're hitting some bad behavior in NFS v3. The situation is this:
machine A - NFS server
machine B - NFS client (writer)
machine C - NFS client (reader)
(all machines x86 SMP)
machine A exports a directory on ext3 filesystem:
/srv/home 192.168.0.0/24(rw,sync,no_subtree_check)
machines B and C mount that directory normally
mount A:/srv/home /mntpnt
machine B opens a file and writes to it (think a log file)
machine C stats that file, opens it and reads it (think tailing the
log file)
The issue is that machine C will often see large blocks of NULLs
(zeros) in the file. If you do the same read again just after you see the block of NULLs you will see proper the data.
Attached are two simple python programs that demonstrate the problem.
To use them (they will write to a file called test-nfs in CWD):
(on machine B in one window)
python writer.py
(on machine C in another window)
python reader.py
reader.py will die when it sees NULLs in the file. Usually for us
this happens after about 60s (two timeouts I think). The first NULL is
usually either at index 4000 or 8000 depending on the kernel.
Now the version of the kernel the server is running doesn't seem to
matter. The reader also doesn't seem to matter (though I didn't test this completely). The writer seems to be the issue:
Writer_Version Outcome:
<= 2.6.19 OK
Quoted 1 line
= 2.6.20 BAD
I've tested both vanilla kernel.org kernels and Ubuntu 8.04 kernels.
I can try to bisect between 2.6.19 <-> 2.6.20.
Anyone else hitting this problem? Any better ideas?
Thanks,
=a=
Signature · 3 lines
=================== Aaron Straus [email protected]
#!/usr/bin/env python
import time
import os
def run(file_name):
data = ''
last_data_len = len(data)
last_size = os.stat(file_name).st_size
while True:
st_size = os.stat(file_name).st_size if st_size != last_size:
print 'size changed @ %s' % time.asctime()
data = open(file_name).read()
if len(data) != last_data_len:
print 'new data arrived @ %s' % time.asctime() print repr(data[-50:])
if '\0' in data:
first_index = data.index('\0')
data_fragment = data[first_index-50:first_index+50] print 'Detected NULL @ %d %s' % (first_index, repr(data_fragment)) break
time.sleep(0.250)
last_data_len = len(data)
last_size = st_size
if __name__ == '__main__':
run('test-nfs')#!/usr/bin/env python
import time
def run(file_name):
fp = open(file_name, 'w')
count = 1
while count:
fp.write('meow\n' * 800)
fp.flush()
time.sleep(32)
fp.close()if __name__ == '__main__':
run('test-nfs')[ replacing cc: [email protected] with [email protected], and neil's old address with his current one ]
Quoted 47 lines · Aaron Straus
Hi all,
We're hitting some bad behavior in NFS v3. The situation is this:
machine A - NFS server
machine B - NFS client (writer)
machine C - NFS client (reader)(all machines x86 SMP)
machine A exports a directory on ext3 filesystem:
/srv/home 192.168.0.0/24(rw,sync,no_subtree_check)machines B and C mount that directory normally
mount A:/srv/home /mntpnt
machine B opens a file and writes to it (think a log file)
machine C stats that file, opens it and reads it (think tailing the
log file)The issue is that machine C will often see large blocks of NULLs
(zeros) in the file. If you do the same read again just after you see the block of NULLs you will see proper the data.Attached are two simple python programs that demonstrate the problem.
To use them (they will write to a file called test-nfs in CWD):
(on machine B in one window)
python
writer.py(on machine C in another window)
python
reader.py
reader.pywill die when it sees NULLs in the file. Usually for us
this happens after about 60s (two timeouts I think). The first NULL is
usually either at index 4000 or 8000 depending on the kernel.
Quoted 7 lines
Now the version of the kernel the server is running doesn't seem to
matter. The reader also doesn't seem to matter (though I didn't test this completely). The writer seems to be the issue:
Writer_VersionOutcome:
<= 2.6.19 OKQuoted 1 line
= 2.6.20 BAD
Up to which kernel? Recent ones may address this issue already.
Quoted 3 lines
I've tested both vanilla kernel.org kernels and Ubuntu 8.04 kernels.
I can try to bisect between 2.6.19 <-> 2.6.20.
That's a good start.
Comparing a wire trace with strace output, starting with the writing client, might also be illuminating. We prefer wireshark as it uses good default trace settings, parses the wire bytes and displays them coherently, and allows you to sort the frames in various useful ways.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hi,
Quoted 2 lines · Chuck Lever
[ replacing cc: [email protected] with [email protected], and neil's old address with his current one ]
Sorry I probably grabbed an old MAINTAINERS file.
Quoted 6 lines · Aaron Straus
Quoted 3 lines
Writer_VersionOutcome:
<= 2.6.19 OKQuoted 1 line
= 2.6.20 BAD
Up to which kernel? Recent ones may address this issue already.
BAD up to 2.6.27-rc?
I have to see exactly which is the last rc version I tested.
Quoted 3 lines
Quoted 1 line
I can try to bisect between 2.6.19 <-> 2.6.20.
That's a good start.
OK will try to bisect.
Quoted 4 lines
Comparing a wire trace with strace output, starting with the writing client, might also be illuminating. We prefer wireshark as it uses good default trace settings, parses the wire bytes and displays them coherently, and allows you to sort the frames in various useful ways.
OK. Could you also try to reproduce on your side using those python programs? I want to make sure it's not something specific with our mounts, etc.
Thanks!
=a=
Signature · 3 lines
=================== Aaron Straus [email protected]
Quoted 25 lines · Aaron Straus
Hi,
Quoted 2 lines · Chuck Lever
[ replacing cc: [email protected] with [email protected], and neil's old address with his current one ]
Sorry I probably grabbed an old
MAINTAINERSfile.Quoted 6 lines · Aaron Straus
Quoted 3 lines
Writer_VersionOutcome:
<= 2.6.19 OKQuoted 1 line
= 2.6.20 BAD
Up to which kernel? Recent ones may address this issue already.
BAD up to 2.6.27-rc?
I have to see exactly which is the last rc version I tested.
Quoted 3 lines
Quoted 1 line
I can try to bisect between 2.6.19 <-> 2.6.20.
That's a good start.
OK will try to bisect.
Quoted 8 lines
Quoted 4 lines
Comparing a wire trace with strace output, starting with the writing client, might also be illuminating. We prefer wireshark as it uses good default trace settings, parses the wire bytes and displays them coherently, and allows you to sort the frames in various useful ways.
OK. Could you also try to reproduce on your side using those python programs? I want to make sure it's not something specific with our mounts, etc.
I have the latest Fedora 9 kernels on two clients, mounting via NFSv3 using "actimeo=600" (for other reasons). The server is OpenSolaris 2008.5.
reader.py reported zeroes in the test file after about 5 minutes.
Looking at the file a little later, I don't see any problems with it.
Since your scripts are not using any kind of serialization (ie file locking) between the clients, I wonder if non-determinant behavior is to be expected.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hi,
Quoted 5 lines · Chuck Lever
I have the latest Fedora 9 kernels on two clients, mounting via NFSv3 using "actimeo=600" (for other reasons). The server is OpenSolaris 2008.5.
reader.pyreported zeroes in the test file after about 5 minutes.
Awesome. Thanks for testing! Our actime is much shorter which is probably why it happens sooner for us.
Quoted 5 lines
Looking at the file a little later, I don't see any problems with it.
Since your scripts are not using any kind of serialization (ie file locking) between the clients, I wonder if non-determinant behavior is to be expected.
Hmm... yep. I don't know what guarantees we want to make. The
behavior doesn't seem to be consistent with older kernels though... so I'm thinking it might be a bug.
We hit this particular issue because we have scripts which essentially
'tail -f' log files looking for errors. They miss log messages (and
see corrupted ones) b/c of the NULLs. That's also why there is no serialization.... we don't need it when grep'ing through log messages.
I'm bisecting now. I see a block of intricate-looking NFS patches, I'll try to narrow it down to a particular commit.
I'll also get the wireshark data at that point.
Thanks, =a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 3 lines · In article <[email protected]> you wrote:
OK. Could you also try to reproduce on your side using those python programs? I want to make sure it's not something specific with our mounts, etc.
sounds like the inode sync comes before the data sync. I suspect thats a client side "problem". Maybe a fsync on the dir or something while the file is open?
Bernd
Quoted 3 lines · Chuck Lever
Quoted 1 line
I can try to bisect between 2.6.19 <-> 2.6.20.
That's a good start.
Hi,
OK. Bisected.
This is the commit where we start to see blocks of NULLs in NFS files.
e261f51f25b98c213e0b3d7f2109b117d714f69d is first bad commit commit e261f51f25b98c213e0b3d7f2109b117d714f69d
Author: Trond Myklebust <[email protected]> Date: Tue Dec 5 00:35:41 2006 -0500
NFS: Make nfs_updatepage() mark the page as dirty.
This will ensure that we can call set_page_writeback() from within nfs_writepage(), which is always called with the page lock set.
Thanks,
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Hi,
Quoted 4 lines · Chuck Lever
Comparing a wire trace with strace output, starting with the writing client, might also be illuminating. We prefer wireshark as it uses good default trace settings, parses the wire bytes and displays them coherently, and allows you to sort the frames in various useful ways.
OK in addition to the bisection I've collected trace data for the good (commit 4d770ccf4257b23a7ca2a85de1b1c22657b581d8) and bad (commit e261f51f25b98c213e0b3d7f2109b117d714f69d) cases.
Attached is a file called trace.tar.bz2 inside you'll find 4 files, for the two sessions:
bad-wireshark
bad-strace
good-wireshark
good-strace
Quoted 1 line
From a quick glance the difference seems to be the bad case does an
UNSTABLE NFS WRITE call. I don't really know what that means or what its semantics are... but that bad commit does seem to introduce this regression.
Anything else I can provide?
Thanks!
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 16 lines · Aaron Straus
Quoted 4 lines · Chuck Lever
Comparing a wire trace with strace output, starting with the writing client, might also be illuminating. We prefer wireshark as it uses good default trace settings, parses the wire bytes and displays them coherently, and allows you to sort the frames in various useful ways.
OK in addition to the bisection I've collected trace data for the good (commit
4d770ccf4257b23a7ca2a85de1b1c22657b581d8) and bad (commite261f51f25b98c213e0b3d7f2109b117d714f69d) cases.Attached is a file called trace.tar.bz2 inside you'll find 4 files, for the two sessions:
bad-wireshark
bad-strace
Quoted 7 lines
good-wireshark
good-straceFrom a quick glance the difference seems to be the bad case does an UNSTABLE NFS WRITE call. I don't really know what that means or what its semantics are... but that bad commit does seem to introduce this regression.
A little analysis reveals the bad behavior on the wire. I filtered the network traces for just WRITE requests. There were six in the good trace, and five in the bad. This seems to correspond with the strace logs, which show that you stopped the bad run before it generated more writes.
Good:
time: offset: len: sync: byte range: 35.402734 0 8000 FILE_SYNC [0, 7999] 100.405170 4096 11904 FILE_SYNC [4096, 15999] 160.407822 12288 7712 FILE_SYNC [12288, 19999] 195.411035 16384 11616 FILE_SYNC [16384, 27999] 260.413574 24576 11424 FILE_SYNC [24576, 35999] 295.704044 32768 7232 FILE_SYNC [32768, 39999]
Bad:
time: offset: len: sync: byte range: 36.989907 0 8000 FILE_SYNC [0, 7999] 101.991164 4096 11904 FILE_SYNC [4096, 15999] 161.992447 12288 7712 FILE_SYNC [12288, 19999] 166.992203 20480 3520 UNSTABLE [20480, 23999] 169.181642 16384 4096 FILE_SYNC [16384, 20479]
The first thing I notice is that the client is re-writing areas of the file. It seems to want to generate page aligned WRITEs, but this means it writes over bytes that were already sent to the server. Probably not the issue here, but that's certainly not optimal behavior. The strace logs show that the application generates 4000 byte write(2) calls, without any operation interleaved that might change the seek offset.
I might go so far to say that this is incorrect behavior as multiple clients writing to a file not on page boundaries would clobber each other's WRITEs.
The second thing is that UNSTABLE WRITE. The fact that it is UNSTABLE is not the problem here. The byte range is what is most interesting:
it leaves a hole in the file, which is fixed by the next write (3 seconds later) which fills in that hole.
Normally this isn't a big deal, as NFS likes applications to serialize access to shared files. However, in this case we expect the .flush() to force all dirty bytes out at once, and this is definitely not occurring....
The third thing is that I don't see any flush or fsync system calls in the strace logs, even though the writer.py script invokes fp.flush(). I'm not terribly familiar with Python's flush(). How do you expect the flush() method to work?
I wonder why the client is using FILE_SYNC writes and not UNSTABLE+COMMIT. Are the clients using the "sync" mount option? I don't see O_SYNC on the open.
open("test-nfs", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
I think I saw some recent work in Trond's development branch that makes some changes in this area. I will wait for him to respond to this thread.
--
Chuck Lever
Hi Chuck,
Quoted 2 lines · Chuck Lever
This seems to correspond with the strace logs, which show that you stopped the bad run before it generated more writes.
Yes. Sorry. I stopped the writer once the reader saw the NULL bytes.
Quoted 9 lines
Normally this isn't a big deal, as NFS likes applications to serialize access to shared files. However, in this case we expect the .flush() to force all dirty bytes out at once, and this is definitely not occurring....
The third thing is that I don't see any flush or fsync system calls in the strace logs, even though the
writer.pyscript invokesfp.flush(). I'm not terribly familiar with Python'sflush(). How do you expect theflush()method to work?
I think fp.flush() in Python is equivalent to fflush() i.e. it only flushes the stdio buffer. It doesn't actually call fsync or fdatasync or anything like that.
We can pretty easily change the Python code to use the syscall write()/etc methods.
Quoted 3 lines
I wonder why the client is using
FILE_SYNCwrites and not UNSTABLE+COMMIT. Are the clients using the "sync" mount option? I don't seeO_SYNCon the open.
I've exported them on the server as sync e.g. in exports:
/export 192.168.0.0/24(rw,sync)
However I don't give any special options when mounting e.e. on the client:
mount machine:/export mntpnt
Anyway thanks for looking at this and let me know if I can do anything else!
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Hi,
Quoted 3 lines · Chuck Lever
I think I saw some recent work in Trond's development branch that makes some changes in this area. I will wait for him to respond to this thread.
One other piece of information.
Of the bisected offending commit:
commit e261f51f25b98c213e0b3d7f2109b117d714f69d
Author: Trond Myklebust <[email protected]> Date: Tue Dec 5 00:35:41 2006 -0500
NFS: Make nfs_updatepage() mark the page as dirty.
This will ensure that we can call set_page_writeback() from within nfs_writepage(), which is always called with the page lock set.
It seems to be this hunk which introduces the problem:
@@ -628,7 +667,6 @@ static struct nfs_page * nfs_update_request(struct nfs_open_context* ctx,
return ERR_PTR(error);
}
spin_unlock(&nfsi->req_lock);
- nfs_mark_request_dirty(new);
return new;
}
spin_unlock(&nfsi->req_lock);
If I add that function call back in... the problem disappears. I don't know if this just papers over the real problem though?
Thanks, =a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 38 lines · Aaron Straus
Hi,
Quoted 3 lines · Chuck Lever
I think I saw some recent work in Trond's development branch that makes some changes in this area. I will wait for him to respond to this thread.
One other piece of information.
Of the bisected offending commit:
commit
e261f51f25b98c213e0b3d7f2109b117d714f69dAuthor: Trond Myklebust <[email protected]> Date: Tue Dec 5 00:35:41 2006 -0500NFS: Make
nfs_updatepage()mark the page as dirty.This will ensure that we can call
set_page_writeback()from withinnfs_writepage(), which is always called with the page lock set.Signed-off-by Trond Myklebust <[email protected]>It seems to be this hunk which introduces the problem:
(quoted diff) +0
-1@@ -628,7 +667,6 @@ static struct nfs_page * nfs_update_request(struct nfs_open_context* ctx, return ERR_PTR(error); } spin_unlock(&nfsi->req_lock); - nfs_mark_request_dirty(new); return new; }spin_unlock(&nfsi->req_lock);
Quoted 2 lines
If I add that function call back in... the problem disappears. I don't know if this just papers over the real problem though?
It's possible that the NFS write path as it exists after this commit is not marking certain pages dirty at the same time as others. An asynchronous flush call from the VM would catch some of the dirty pages, then later, others, resulting in the pages flushing to the server out of order.
However, I poked around a little bit in late model kernels yesterday and found that the changes from Trond I referred to earlier are already in 2.6.27. These rework this area significantly, so reverting this hunk alone will be more difficult in recent kernels.
In addition I notice that one purpose of this commit is to change the page locking scheme in the NFS client's write path. I wouldn't advise simply adding this nfs_mark_request_dirty() call back in. It could result in deadlocks for certain workloads (perhaps not your specific workload, though).
A more thorough review of the NFS write and flush logic that exists in 2.6.27 is needed if we choose to recognize this issue as a real problem.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hi,
Quoted 3 lines · Chuck Lever
A more thorough review of the NFS write and flush logic that exists in 2.6.27 is needed if we choose to recognize this issue as a real problem.
Yep.
Sorry. I didn't mean we should revert the hunk. I was just trying to help identify the cause of the new behavior.
I think this is a real problem albeit not a "serious" one. Network file-systems usually try to avoid readers seeing blocks of zeros in files, especially in this simple writer/reader case.
It wouldn't be bad if the file is written out of order occasionally, but we see this constantly now.
We cannot write/read log files to NFS mounts reliably any more. That seems like a valid use case which no longer works?
Anyway, thanks for your time!
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 16 lines · Aaron Straus
Hi,
Quoted 3 lines · Chuck Lever
A more thorough review of the NFS write and flush logic that exists in 2.6.27 is needed if we choose to recognize this issue as a real problem.
Yep.
Sorry. I didn't mean we should revert the hunk. I was just trying to help identify the cause of the new behavior.
I think this is a real problem albeit not a "serious" one. Network file-systems usually try to avoid readers seeing blocks of zeros in files, especially in this simple writer/reader case.
Quoted 5 lines
It wouldn't be bad if the file is written out of order occasionally, but we see this constantly now.
We cannot write/read log files to NFS mounts reliably any more. That seems like a valid use case which no longer works?
Were you able to modify your writer to do real fsync system calls? If so, did it help? That would be a useful data point.
NFS uses close-to-open cache coherency. Thus we expect the file to be consistent and up to date after it is opened by a client, but not necessarily if some other client writes to it after it was opened. We usually recommend file locking and direct I/O to minimize these problems.
Practically speaking this is often not enough for typical applications, so NFS client implementations go to further (non-standard) efforts to behave like a local file system. This is simply a question of whether we can address this while not creating performance or correctness issues for other common use cases.
Anyway, I'm not the NFS client maintainer, so this decision is not up to me.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Hi,
Quoted 2 lines · Chuck Lever
Were you able to modify your writer to do real fsync system calls? If so, did it help? That would be a useful data point.
Yes/Yes. Please see the attached tarball sync-test.tar.bz2.
Inside you'll find the modified writer: writer_sync.py
That will call fsync on the file descriptor after each write.
Also I added the strace and wireshark data for the sync and nosync cases.
Note these are all tested with latest linus git:
d1c6d2e547148c5aa0c0a4ff6aac82f7c6da1d8b
Quoted 5 lines
Practically speaking this is often not enough for typical applications, so NFS client implementations go to further (non- standard) efforts to behave like a local file system. This is simply a question of whether we can address this while not creating performance or correctness issues for other common use cases.
Yep, I agree. I'm not saying what we do now is "wrong" per the RFC (writing the file out of order). It's just different from what we've done in the past (and somewhat unexpected).
I'm still hoping there is a simple fix... but maybe not... :(
Thanks!
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Hi Aaron, hi NFS hackers,
Am Donnerstag, 11. September 2008 schrieb Aaron Straus:
Quoted 25 lines
Hi,
Quoted 2 lines · Chuck Lever
Were you able to modify your writer to do real fsync system calls? If so, did it help? That would be a useful data point.
Yes/Yes. Please see the attached tarball sync-test.tar.bz2.
Inside you'll find the modified writer:
writer_sync.pyThat will call fsync on the file descriptor after each write.
Also I added the strace and wireshark data for the sync and nosync cases.
Note these are all tested with latest linus git:
d1c6d2e547148c5aa0c0a4ff6aac82f7c6da1d8bQuoted 5 lines
Practically speaking this is often not enough for typical applications, so NFS client implementations go to further (non- standard) efforts to behave like a local file system. This is simply a question of whether we can address this while not creating performance or correctness issues for other common use cases.
Quoted 5 lines
Yep, I agree. I'm not saying what we do now is "wrong" per the RFC (writing the file out of order). It's just different from what we've done in the past (and somewhat unexpected).
I'm still hoping there is a simple fix... but maybe not... :(
For what is worth, this behavior is visible in bog standard writing/reading files, (log files in my case, via the python logging package). It obviously deviates from local filesystem behavior, and former state of the linux nfs-client. Should we add patches to less, tail, and all other instruments for watching/analysing log files (just to pick the tip of the ice rock) in order to throw away runs of zeros, when reading from nfs mounted files? Or should we ask their maintainers to add locking code for the nfs "read files, which are written at the same time" case, just to work around __some__ of the consequences of this bug? Imagine, how ugly this is going to look!
The whole issue is what I call a major regression, thus I strongly ask for a reply from Trond on this matter.
I even vote for sending a revert request for this hunk to the stable team, where it is applicable, after Trond sorted it out (for 2.6.27?).
Thanks, Aaron and Chuck for the detailed analysis - it demystified a wired behavior, I observed here. When you're in a process to get real work done in a fixed timeline, such things could make you mad..
Cheers,
Pete
Quoted 14 lines · Hans-Peter Jansen
For what is worth, this behavior is visible in bog standard writing/reading files, (log files in my case, via the python logging package). It obviously deviates from local filesystem behavior, and former state of the linux nfs-client. Should we add patches to less, tail, and all other instruments for watching/analysing log files (just to pick the tip of the ice rock) in order to throw away runs of zeros, when reading from nfs mounted files? Or should we ask their maintainers to add locking code for the nfs "read files, which are written at the same time" case, just to work around
__some__of the consequences of this bug? Imagine, how ugly this is going to look!The whole issue is what I call a major regression, thus I strongly ask for a reply from Trond on this matter.
Quoted 6 lines
I even vote for sending a revert request for this hunk to the stable team, where it is applicable, after Trond sorted it out (for 2.6.27?).
Thanks, Aaron and Chuck for the detailed analysis - it demystified a wired behavior, I observed here. When you're in a process to get real work done in a fixed timeline, such things could make you mad..
Revert _what_ exactly?
Please assume that I've been travelling for the past 5 weeks, and have only a sketchy idea of what has been going on.
My understanding was that this is a consequence of unordered writes causing the file to be extended while some other task is reading. AFAICS, this sort of behaviour has _always_ been possible. I can't see how reverting anything will fix it.
Trond
Hi,
Quoted 1 line · Trond Myklebust
Revert _what_ exactly?
Yep. I narrowed the problem down to an offending hunk in a particular
patch. Removing that hunk did eliminate the problem. However,
reverting that hunk is likely wrong and the code has changed _a lot_ since that commit.
Quoted 2 lines
My understanding was that this is a consequence of unordered writes causing the file to be extended while some other task is reading.
Yes. I added some debugging statements to look at the writeout path.
I think the following happens:
- page 0 gets partially written to by app
- VM writes out partial dirty page 0
- page 0 gets fully written by app
- page 1 gets partially written by app
- VM writes out dirty page 1
At this point there is a hole in the file. The tail end of page 0 is
still not written to server.
- VM writes out dirty page 0
...
Quoted 2 lines
AFAICS, this sort of behaviour has _always_ been possible. I can't see how reverting anything will fix it.
Here is the crux. It was possible previously but unlikely e.g. our app never saw this behavior. The new writeout semantics causes visible holes in files often.
Anyway, I agree the new writeout semantics are allowed and possibly saner than the previous writeout path. The problem is that it is __annoying__ for this use case (log files).
I'm not sure if there is an easy solution. We want the VM to writeout
the address space in order. Maybe we can start the scan for dirty
pages at the last page we wrote out i.e. page 0 in the example above?
=a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 36 lines · Aaron Straus
Hi,
Quoted 1 line · Trond Myklebust
Revert _what_ exactly?
Yep. I narrowed the problem down to an offending hunk in a particular
patch. Removing that hunk did eliminate the problem. However,
reverting that hunk is likely wrong and the code has changed _alot_since that commit.Quoted 2 lines
My understanding was that this is a consequence of unordered writes causing the file to be extended while some other task is reading.
Yes. I added some debugging statements to look at the writeout path.
I think the following happens:
- page 0 gets partially written to by app
- VM writes out partial dirty page 0
- page 0 gets fully written by app
- page 1 gets partially written by app
- VM writes out dirty page 1At this point there is a hole in the file. The tail end of page 0 is
still not written to server.- VM writes out dirty page 0
...Quoted 2 lines
AFAICS, this sort of behaviour has _always_ been possible. I can't see how reverting anything will fix it.
Here is the crux. It was possible previously but unlikely e.g. our app never saw this behavior. The new writeout semantics causes visible holes in files often.
Quoted 7 lines
Anyway, I agree the new writeout semantics are allowed and possibly saner than the previous writeout path. The problem is that it is
__annoying__for this use case (log files).I'm not sure if there is an easy solution. We want the VM to writeout
the address space in order. Maybe we can start the scan for dirty
pages at the last page we wrote out i.e. page 0 in the example above?
Why can't you use O_SYNC | O_APPEND ?
--
Chuck Lever
Hi,
Quoted 1 line · Chuck Lever
Why can't you use
O_SYNC|O_APPEND?
In our case, some of the writers are not in our control. Another case we see the issue is you spawn a job:
job 1> out 2> err &
tail -f out err
--
Also, we actually do like caching the writes (b/c log files do not need
to be checked immediately after being written). We just wish the cache
could be written out in-order.
If there's no way to make that happen... we can reorganize our file-system, exports, mounts, etc so that the log file directory is mounted with the sync option.
However, the problem *might* be annoying and wide-spread enough to warrant a change.
Anyway, thanks!
Regards, =a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Quoted 7 lines · Aaron Straus
Here is the crux. It was possible previously but unlikely e.g. our app never saw this behavior. The new writeout semantics causes visible holes in files often.
Anyway, I agree the new writeout semantics are allowed and possibly saner than the previous writeout path. The problem is that it is
__annoying__for this use case (log files).
There is always the option of using syslog.
Quoted 3 lines
I'm not sure if there is an easy solution. We want the VM to writeout
the address space in order. Maybe we can start the scan for dirty
pages at the last page we wrote out i.e. page 0 in the example above?
You can never guarantee that in a multi-threaded environment.
Two threads may, for instance, force 2 competing fsync() calls: that again may cause out-of-order writes.
...and even if the client doesn't reorder the writes, the _server_ may do it, since multiple nfsd threads may race when processing writes to the same file.
Anyway, the patch to force a single threaded nfs client to write out the data in order is trivial. See attachment...
Trond
---
fs/nfs/write.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
fs/nfs/write.c +2 -1
@@ -1428,7 +1428,8 @@ static int nfs_write_mapping(struct address_space *mapping, int how) .sync_mode = WB_SYNC_NONE, .nr_to_write = LONG_MAX, .for_writepages = 1, - .range_cyclic = 1, + .range_start = 0, + .range_end = LLONG_MAX, };int ret;
Hi,
Quoted 5 lines · Trond Myklebust
Quoted 3 lines
Anyway, I agree the new writeout semantics are allowed and possibly saner than the previous writeout path. The problem is that it is
__annoying__for this use case (log files).There is always the option of using syslog.
Definitely. Everything in our control we can work around.... there are a few applications we cannot easily change... see the follow-up in another e-mail.
Quoted 5 lines
Quoted 3 lines
I'm not sure if there is an easy solution. We want the VM to writeout
the address space in order. Maybe we can start the scan for dirty
pages at the last page we wrote out i.e. page 0 in the example above?You can never guarantee that in a multi-threaded environment.
Definitely. This is a single writer, single reader case though.
Quoted 2 lines
Two threads may, for instance, force 2 competing
fsync()calls: that again may cause out-of-order writes.
Yup.
Quoted 3 lines
...and even if the client doesn't reorder the writes, the _server_ may do it, since multiple nfsd threads may race when processing writes to the same file.
Yup. We're definitely not asking for anything like that.
Quoted 8 lines
Anyway, the patch to force a single threaded nfs client to write out the data in order is trivial. See attachment...
fs/nfs/write.c +0
-0
fs/nfs/write.c static int nfs_write_mapping(struct address_space *mapping, int how) · line 1435
Quoted 9 lines
fs/nfs/write.c +2
-1@@ -1428,7 +1428,8 @@ static int nfs_write_mapping(struct address_space *mapping, int how) .sync_mode = WB_SYNC_NONE, .nr_to_write = LONG_MAX, .for_writepages = 1, - .range_cyclic = 1, + .range_start = 0, + .range_end = LLONG_MAX, };int ret;
Yeah I was looking at that while debugging. Would that change have chance to make it into mainline? I assume it makes the normal writeout path more expensive, by forcing a scan of the entire address space.
Also, I should test this, but I thought the VM was calling nfs_writepages directly i.e. not going through nfs_write_mapping. Let me test with this patch.
Thanks, =a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Hi,
Quoted 10 lines · Aaron Straus
Quoted 10 lines
fs/nfs/write.c +1
-1@@ -1428,7 +1428,8 @@ static int nfs_write_mapping(struct address_space *mapping, int how) .sync_mode = WB_SYNC_NONE, .nr_to_write = LONG_MAX, .for_writepages = 1, - .range_cyclic = 1, + .range_start = 0,
Quoted 8 lines
fs/nfs/write.c static int nfs_write_mapping(struct address_space *mapping, int how) · line 1435
Quoted 4 lines
fs/nfs/write.c +1
-0+ .range_end = LLONG_MAX, };int ret;Also, I should test this, but I thought the VM was calling
nfs_writepagesdirectly i.e. not going throughnfs_write_mapping. Let me test with this patch.
Yes. This patch doesn't seem to help. The VM is calling nfs_writepages
directly. I have debug statements in the nfs_write_mapping() call and it
hasn't been called when the hole appears.
It's the VM dirty page writeout that is happening out of order, *not*
nfs_wb_* stuff. That's why we can see it so easily. The dirty pages
are only scanned periodically. In-between scans is when we see the hole.
I think :). I'm obviously not an expert here.
Thanks, =a=
Signature · 3 lines
=================== Aaron Straus aaron-bYFJunmd+ZV8UrSeD/[email protected]
Am Montag, 22. September 2008 schrieb Aaron Straus:
Quoted 13 lines
Hi,
Quoted 5 lines · Trond Myklebust
Quoted 3 lines
Anyway, I agree the new writeout semantics are allowed and possibly saner than the previous writeout path. The problem is that it is
__annoying__for this use case (log files).There is always the option of using syslog.
Definitely. Everything in our control we can work around.... there are a few applications we cannot easily change... see the follow-up in another e-mail.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I'm not sure if there is an easy solution. We want the VM to
writeout the address space in order. Maybe we can start the scan
for dirty pages at the last page we wrote out i.e. page 0 in the example above?You can never guarantee that in a multi-threaded environment.
Definitely. This is a single writer, single reader case though.
...where it happens, that the reader gets chunks of zeros from reading a file, that is written from another (single threaded) process.
Note, that going through syslog isn't an option in many cases unless we want to rewrite the "world" to work around this phenomenon, thus it's not simply annoying, as Aaron points out, the "in order" approach is inevitable.
Quoted 23 lines
Quoted 2 lines
Two threads may, for instance, force 2 competing
fsync()calls: that again may cause out-of-order writes.Yup.
Quoted 3 lines
...and even if the client doesn't reorder the writes, the _server_ may do it, since multiple nfsd threads may race when processing writes to the same file.
Yup. We're definitely not asking for anything like that.
Quoted 12 lines
Anyway, the patch to force a single threaded nfs client to write out the data in order is trivial. See attachment...
fs/nfs/write.c +0
-1@@ -1428,7 +1428,8 @@ static int nfs_write_mapping(struct address_space *mapping, int how) .sync_mode = WB_SYNC_NONE, .nr_to_write = LONG_MAX, .for_writepages = 1, - .range_cyclic = 1,
Quoted 8 lines
fs/nfs/write.c static int nfs_write_mapping(struct address_space *mapping, int how) · line 1434
Quoted 4 lines
fs/nfs/write.c +2
-0+ .range_start = 0, + .range_end = LLONG_MAX, };int ret;Yeah I was looking at that while debugging. Would that change have chance to make it into mainline? I assume it makes the normal writeout path more expensive, by forcing a scan of the entire address space.
If this patch solves this issue, it is necessary to get applied as soon as possible as outlined above..
Quoted 3 lines
Also, I should test this, but I thought the VM was calling
nfs_writepagesdirectly i.e. not going throughnfs_write_mapping. Let me test with this patch.
Let us know about the outcome.
Thanks,
Pete
Am Montag, 22. September 2008 schrieb Trond Myklebust:
Quoted 15 lines · Hans-Peter Jansen
Quoted 14 lines
For what is worth, this behavior is visible in bog standard writing/reading files, (log files in my case, via the python logging package). It obviously deviates from local filesystem behavior, and former state of the linux nfs-client. Should we add patches to less, tail, and all other instruments for watching/analysing log files (just to pick the tip of the ice rock) in order to throw away runs of zeros, when reading from nfs mounted files? Or should we ask their maintainers to add locking code for the nfs "read files, which are written at the same time" case, just to work around
__some__of the consequences of this bug? Imagine, how ugly this is going to look!The whole issue is what I call a major regression, thus I strongly ask for a reply from Trond on this matter.
Quoted 8 lines
Quoted 6 lines
I even vote for sending a revert request for this hunk to the stable team, where it is applicable, after Trond sorted it out (for 2.6.27?).
Thanks, Aaron and Chuck for the detailed analysis - it demystified a wired behavior, I observed here. When you're in a process to get real work done in a fixed timeline, such things could make you mad..
Revert _what_ exactly?
For your convenience, important parts inlined here:
Quoted 1 line
From Aarons message: Tue, 9 Sep 2008 12:46:44 -0700 in this thread. << EOM
Of the bisected offending commit:
commit e261f51f25b98c213e0b3d7f2109b117d714f69d
Author: Trond Myklebust <[email protected]> Date: Tue Dec 5 00:35:41 2006 -0500
NFS: Make nfs_updatepage() mark the page as dirty.
This will ensure that we can call set_page_writeback() from within nfs_writepage(), which is always called with the page lock set.
It seems to be this hunk which introduces the problem:
@@ -628,7 +667,6 @@ static struct nfs_page * nfs_update_request(struct nfs_open_context* ctx,
return ERR_PTR(error);
}
spin_unlock(&nfsi->req_lock);
- nfs_mark_request_dirty(new);
return new;
}
spin_unlock(&nfsi->req_lock);If I add that function call back in... the problem disappears. I don't know if this just papers over the real problem though?
EOM
This commit happened between 2.6.19 and 2.6.20, btw.
Quoted 2 lines
Please assume that I've been travelling for the past 5 weeks, and have only a sketchy idea of what has been going on.
Ahh, I see, that explains, why you didn't responded earlier.
Quoted 4 lines
My understanding was that this is a consequence of unordered writes causing the file to be extended while some other task is reading. AFAICS, this sort of behaviour has _always_ been possible. I can't see how reverting anything will fix it.
Hopefully, this helps you to remember the purpose of that change.
Cheers,
Pete