another nfs puzzle
Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.
The attached file demonstrates this.
Compile with -DABUSE to trigger the bad case. This behavior does not happen with an ext3 partition.
ethereal shows the behavior to be a large amount of block reads, with single byte writes every so often. Viewing the resultant file from other hosts or even the original host shows the file is grown, but is zero-filled, not 'a'-filled.
-Kenny
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
#define _GNU_SOURCE
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
/*
* the 'abusive' case is where we
* grow the file via pwrite * unmap the previous mapping, * do the new mapping * * the non-abusive case we * unmap the previous mapping * grow the file * do the new mapping
*
* the difference being that in the abusive case,
* we maintain a mapping during the pwrite.
*
* If we grow the file via pwrite and maintain a
* mapping, the mapped pages are never written out. *
* If we either grow the file via ftruncate or remove
* old mappings before growing the file, then all is
* fine.
*/
#ifdef ABUSE
#define UNMAP_AFTER() munmap(mapping, size)
#define UNMAP_BEFORE() struct eat_semi
#else
#define UNMAP_AFTER() struct eat_semi
#define UNMAP_BEFORE() munmap(mapping, size)
#endif
int main(int argc, char* argv[])
{
int fd;
unsigned int const size = 4096 * 1024;
unsigned int offset = 0;
if (argc != 2) {
printf("usage: %s <filename>\n", argv[0]);
return 0;
}
fd = open(argv[1], O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE | O_DIRECT, 0644);
if (fd < 0) {
perror("open");
return 0;
}
pwrite64(fd, "", 1, offset + size);
//ftruncate64(fd, offset + size);
char* mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
memset(mapping, 'a', size);
UNMAP_BEFORE();
for (;;) {
offset += size; pwrite64(fd, "", 1, offset + size);
//ftruncate64(fd, offset + size);
UNMAP_AFTER();
mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
memset(mapping, 'a', size);
UNMAP_BEFORE();}
close(fd);
return 0;
}
Quoted 5 lines · Kenny Simpson
Hi again,
I am seeing some odd behavior withO_DIRECT. If a file opened withO_DIRECThas a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.
Does the attached patch fix it?
Cheers,
Trond
NFS: Fix another O_DIRECT race
Ensure we call unmap_mapping_range() and sync dirty pages to disk before doing an NFS direct write.
---
fs/nfs/direct.c | 43 ++++++++++++++++++++++++------------------- 1 files changed, 24 insertions(+), 19 deletions(-)
fs/nfs/direct.c +24 -19
@@ -111,6 +111,21 @@ nfs_get_user_pages(int rw, unsigned longreturn result; } +static int nfs_sync_mapping (struct address_space *mapping,loff_t offset,size_t len) +{ +int ret; +if (mapping->nrpages) +return 0 ; + +unmap_mapping_range (mapping, offset, len,0 ); + ret =filemap_fdatawrite (mapping); +if (ret ==0 ) + ret =filemap_fdatawait (mapping); +if (ret ==0 ) + ret =nfs_wb_all (mapping->host); +return ret; +} +/** * nfs_free_user_pages - tear down page struct array * @pages: array of page struct pointers underlying target buffer
@@ -676,15 +691,9 @@ nfs_file_direct_read(struct kiocb *iocb,if (!count) goto out; -if (mapping->nrpages) { - retval =filemap_fdatawrite (mapping); -if (retval ==0 ) - retval =nfs_wb_all (inode); -if (retval ==0 ) - retval =filemap_fdatawait (mapping); -if (retval) - goto out; - } + retval =nfs_sync_mapping (mapping, pos, count); +if (retval) + goto out; retval =nfs_direct_read (inode, ctx, &iov, pos,1 );if (retval >0 )
@@ -762,19 +771,15 @@ nfs_file_direct_write(struct kiocb *iocbif (!count) goto out; -if (mapping->nrpages) { - retval =filemap_fdatawrite (mapping); -if (retval ==0 ) - retval =nfs_wb_all (inode); -if (retval ==0 ) - retval =filemap_fdatawait (mapping); -if (retval) - goto out; - } + retval =nfs_sync_mapping (mapping, pos, count); +if (retval) + goto out; retval =nfs_direct_write (inode, ctx, &iov, pos,1 );if (mapping->nrpages) -invalidate_inode_pages2 (mapping); +invalidate_inode_pages2_range (mapping, + pos >>PAGE_CACHE_SHIFT , + (pos + count -1 ) >>PAGE_CACHE_SHIFT );if (retval >0 ) *ppos = pos + retval;
Quoted 1 line · --- Trond Myklebust
Does the attached patch fix it?
Dude, you're a machine! Yes, it fixes it.
-Kenny
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
Quoted 6 lines · Kenny Simpson
Hi again,
I am seeing some odd behavior withO_DIRECT. If a file opened withO_DIRECThas a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.
I don't think that I understand why or how the kernel allows a file, which was opened with O_DIRECT, to be mmap'd. The use of O_DIRECT implies no caching and mmap implies the use of caching.
I do understand that the kernel can flush and invalidate pages in the ranges of the i/o operations done, but does it really guarantee that a pagein operation won't happen on a page within the range of a region of the file being written to using direct i/o? If it does not, then any consistency guarantees are gone and data corruption can occur.
Thanx...
ps
Quoted 9 lines · Peter Staubach
Quoted 6 lines · Kenny Simpson
Hi again,
I am seeing some odd behavior withO_DIRECT. If a file opened withO_DIRECThas a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.
Quoted 3 lines
I don't think that I understand why or how the kernel allows a file, which was opened with
O_DIRECT, to be mmap'd. The use ofO_DIRECTimplies no caching and mmap implies the use of caching.
In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for O_DIRECT and then your process open it for normal I/O, and mmaps it.
Cheers,
Trond
Quoted 21 lines · Trond Myklebust
Quoted 12 lines · Peter Staubach
Quoted 8 lines · Kenny Simpson
Hi again,
I am seeing some odd behavior withO_DIRECT. If a file opened withO_DIRECThas a page mmap'd, and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.Quoted 5 lines
I don't think that I understand why or how the kernel allows a file, which was opened with
O_DIRECT, to be mmap'd. The use ofO_DIRECTimplies no caching and mmap implies the use of caching.
Quoted 4 lines
In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for
O_DIRECTand then your process open it for normal I/O, and mmaps it.
Yup, same problem. Why is this allowed? Does it really work correctly?
Thanx...
ps
Quoted 6 lines · Peter Staubach
Quoted 4 lines
In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for
O_DIRECTand then your process open it for normal I/O, and mmaps it.Yup, same problem. Why is this allowed? Does it really work correctly?
Assuming that the processes have _some_ method of synchronisation, then I cannot see why it shouldn't be workable. Come to think of it, it might even be possible to use O_DIRECT to provide that synchronisation (use O_DIRECT to set a "lock" on the page, then modify it using mmap).
Whether or not there are people out there that actually _want_ to do this is a different matter.
Cheers,
Trond
Quoted 4 lines · Trond Myklebust
Quoted 3 lines
Yup, same problem. Why is this allowed? Does it really work correctly?
Quoted 8 lines
Assuming that the processes have _some_ method of synchronisation, then I cannot see why it shouldn't be workable. Come to think of it, it might even be possible to use
O_DIRECTto provide that synchronisation (useO_DIRECTto set a "lock" on the page, then modify it using mmap).Whether or not there are people out there that actually _want_ to do this is a different matter.
Mixing O_DIRECT i/o and cached i/o is probably a recipe for disaster, unless the cooperating programs are very careful and very aware of how the particular file system in the particular kernel implements direct i/o and caching, including cache validation.
This seems like a dangerous enough area that denying mmap on a file which has been opened with O_DIRECT by any process and denying open(O_DIRECT) on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.
Thanx...
ps
Quoted 4 lines · Peter Staubach
This seems like a dangerous enough area that denying mmap on a file which has been opened with
O_DIRECTby any process and denyingopen(O_DIRECT)on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.
That would be a recipe for DOSes as it would allow one process to block another just by opening a file. I'd really not like to see my database crash just because some obscure monitoring program happens to use mmap() to browse the file.
Cheers,
Trond
Quoted 11 lines · Trond Myklebust
Quoted 6 lines · Peter Staubach
This seems like a dangerous enough area that denying mmap on a file which has been opened with
O_DIRECTby any process and denyingopen(O_DIRECT)on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.
Quoted 5 lines
That would be a recipe for DOSes as it would allow one process to block another just by opening a file. I'd really not like to see my database crash just because some obscure monitoring program happens to use
mmap()to browse the file.
I wouldn't think that a database would be a problem since it is opened early and then stays open. However, point granted. There are tools around, lsof and such, which would be handy for diagnosing such a situation though.
I do know of other operating systems which do implement the semantics that I have suggested and I don't think that they are concerned or have been notified that these semantics for a problem. These semantics are used because the kernel itself can not even guarantee that the data that it is caching is valid, without lots of synchronization which may tend to reduce performance.
Thanx...
ps
Quoted 7 lines · Peter Staubach
I do know of other operating systems which do implement the semantics that I have suggested and I don't think that they are concerned or have been notified
that these semantics for a problem. These semantics are used because the kernel itself can not even guarantee that the data that it is caching is valid,
without lots of synchronization which may tend to reduce performance.
I agree. It is clearly going to be a drag on performance, but it should be very much a corner case, so...
To tell you the truth, I'm more interested in this case in order to figure out how to make mmap() work correctly for the case of an ordinary file, but where the file changes on the server. Currently we just call invalidate_inode_pages2(), without unmapping first. The conclusion from Kenny's testcase appears to be that this may lead to mmapped dirty data being lost.
Cheers,
Trond
Quoted 6 lines · Trond Myklebust
I agree. It is clearly going to be a drag on performance, but it should be very much a corner case, so...
Yes, file systems tend to mostly be about the corner cases though... :-)
Quoted 7 lines
To tell you the truth, I'm more interested in this case in order to figure out how to make
mmap()work correctly for the case of an ordinary file, but where the file changes on the server. Currently we just callinvalidate_inode_pages2(), without unmapping first. The conclusion from Kenny's testcase appears to be that this may lead to mmapped dirty data being lost.
Ugly, huh? The options seem to be to either write out all of the data to the server or just truncate it. I have seen the former used mostly, although this does generate the "last one there wins" scenario. This does match the usual semantics associated with normal cached data from write(2)s and should fit well with the NFSv4 callback notifying that a write delegation is being withdrawn.
Thanx...
ps