Skip to contentSkip to conversation

another nfs puzzle

DISCUSSION Kenny Simpson 3 participants 12 messages Dec 6, 2005 – Dec 7, 2005 NFS client

lore
Share & tools
lore
#0 lore raw

Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.

The attached file demonstrates this.
Compile with -DABUSE to trigger the bad case. This behavior does not happen with an ext3 partition.

ethereal shows the behavior to be a large amount of block reads, with single byte writes every so often. Viewing the resultant file from other hosts or even the original host shows the file is grown, but is zero-filled, not 'a'-filled.

-Kenny

__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

#define _GNU_SOURCE

#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#include <stdio.h>
#include <string.h>

/*
* the 'abusive' case is where we

 *   grow the file via pwrite
 *   unmap the previous mapping,
 *   do the new mapping
 *
 * the non-abusive case we
 *   unmap the previous mapping
 *   grow the file
 *   do the new mapping

*
* the difference being that in the abusive case,
* we maintain a mapping during the pwrite.
*
* If we grow the file via pwrite and maintain a
* mapping, the mapped pages are never written out. *
* If we either grow the file via ftruncate or remove
* old mappings before growing the file, then all is
* fine.
*/

#ifdef ABUSE
#define UNMAP_AFTER() munmap(mapping, size)
#define UNMAP_BEFORE() struct eat_semi
#else
#define UNMAP_AFTER() struct eat_semi
#define UNMAP_BEFORE() munmap(mapping, size)
#endif

int main(int argc, char* argv[])
{
int fd;
unsigned int const size = 4096 * 1024;
unsigned int offset = 0;

  if (argc != 2) {
    printf("usage: %s <filename>\n", argv[0]);
    return 0;
  }

  fd = open(argv[1], O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE | O_DIRECT, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

  pwrite64(fd, "", 1, offset + size);
  //ftruncate64(fd, offset + size);
  char* mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
  memset(mapping, 'a', size);
  UNMAP_BEFORE();

  for (;;) {
    offset += size;
    pwrite64(fd, "", 1, offset + size);
    //ftruncate64(fd, offset + size);
    UNMAP_AFTER();
    mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
    memset(mapping, 'a', size);
    UNMAP_BEFORE();

}

  close(fd);

return 0;
}

#1 lore raw
Quoted 5 lines · Kenny Simpson

Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.

Does the attached patch fix it?

Cheers,
Trond

NFS: Fix another O_DIRECT race

Ensure we call unmap_mapping_range() and sync dirty pages to disk before doing an NFS direct write.

---

 fs/nfs/direct.c |   43 ++++++++++++++++++++++++-------------------
 1 files changed, 24 insertions(+), 19 deletions(-)
fs/nfs/direct.c +24 -19
index a2d2814..ce83000 100644
@@ -111,6 +111,21 @@ nfs_get_user_pages(int rw, unsigned long
 	return result;
 }
 
+static int nfs_sync_mapping(struct address_space *mapping, loff_t offset, size_t len)
+{
+	int ret;
+	if (mapping->nrpages)
+		return 0;
+
+	unmap_mapping_range(mapping, offset, len, 0);
+	ret = filemap_fdatawrite(mapping);
+	if (ret == 0)
+		ret = filemap_fdatawait(mapping);
+	if (ret == 0)
+		ret = nfs_wb_all(mapping->host);
+	return ret;
+}
+
 /**
  * nfs_free_user_pages - tear down page struct array
  * @pages: array of page struct pointers underlying target buffer
@@ -676,15 +691,9 @@ nfs_file_direct_read(struct kiocb *iocb,
 	if (!count)
 		goto out;
 
-	if (mapping->nrpages) {
-		retval = filemap_fdatawrite(mapping);
-		if (retval == 0)
-			retval = nfs_wb_all(inode);
-		if (retval == 0)
-			retval = filemap_fdatawait(mapping);
-		if (retval)
-			goto out;
-	}
+	retval = nfs_sync_mapping(mapping, pos, count);
+	if (retval)
+		goto out;
 
 	retval = nfs_direct_read(inode, ctx, &iov, pos, 1);
 	if (retval > 0)
@@ -762,19 +771,15 @@ nfs_file_direct_write(struct kiocb *iocb
 	if (!count)
 		goto out;
 
-	if (mapping->nrpages) {
-		retval = filemap_fdatawrite(mapping);
-		if (retval == 0)
-			retval = nfs_wb_all(inode);
-		if (retval == 0)
-			retval = filemap_fdatawait(mapping);
-		if (retval)
-			goto out;
-	}
+	retval = nfs_sync_mapping(mapping, pos, count);
+	if (retval)
+		goto out;
 
 	retval = nfs_direct_write(inode, ctx, &iov, pos, 1);
 	if (mapping->nrpages)
-		invalidate_inode_pages2(mapping);
+		invalidate_inode_pages2_range(mapping,
+				pos >> PAGE_CACHE_SHIFT,
+				(pos + count - 1) >> PAGE_CACHE_SHIFT);
 	if (retval > 0)
 		*ppos = pos + retval;
 
#2 lore raw
Quoted 1 line · --- Trond Myklebust

Does the attached patch fix it?

Dude, you're a machine! Yes, it fixes it.

-Kenny

__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

#3 lore raw
In reply to Kenny Simpson
Quoted 6 lines · Kenny Simpson

Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.

I don't think that I understand why or how the kernel allows a file, which was opened with O_DIRECT, to be mmap'd. The use of O_DIRECT implies no caching and mmap implies the use of caching.

I do understand that the kernel can flush and invalidate pages in the ranges of the i/o operations done, but does it really guarantee that a pagein operation won't happen on a page within the range of a region of the file being written to using direct i/o? If it does not, then any consistency guarantees are gone and data corruption can occur.

Thanx...

ps

#4 lore raw
Quoted 9 lines · Peter Staubach
Quoted 6 lines · Kenny Simpson

Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd,
and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.

Quoted 3 lines

I don't think that I understand why or how the kernel allows a file, which was opened with O_DIRECT, to be mmap'd. The use of O_DIRECT implies no caching and mmap implies the use of caching.

In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for O_DIRECT and then your process open it for normal I/O, and mmaps it.

Cheers,
Trond

#5 lore raw
Quoted 21 lines · Trond Myklebust
Quoted 12 lines · Peter Staubach
Quoted 8 lines · Kenny Simpson

Hi again,
I am seeing some odd behavior with O_DIRECT. If a file opened with O_DIRECT has a page mmap'd, and the file is extended via pwrite, then the mmap'd region seems to get lost - i.e. it neither takes up system memory, nor does it get written out.

Quoted 5 lines

I don't think that I understand why or how the kernel allows a file, which was opened with O_DIRECT, to be mmap'd. The use of O_DIRECT implies no caching and mmap implies the use of caching.

Quoted 4 lines

In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for O_DIRECT and then your process open it for normal I/O, and mmaps it.

Yup, same problem. Why is this allowed? Does it really work correctly?

Thanx...

ps

#6 lore raw
Quoted 6 lines · Peter Staubach
Quoted 4 lines

In this context it doesn't matter whether or not the you use the same file descriptor. The problem is the same if my process opens the file for O_DIRECT and then your process open it for normal I/O, and mmaps it.

Yup, same problem. Why is this allowed? Does it really work correctly?

Assuming that the processes have _some_ method of synchronisation, then I cannot see why it shouldn't be workable. Come to think of it, it might even be possible to use O_DIRECT to provide that synchronisation (use O_DIRECT to set a "lock" on the page, then modify it using mmap).

Whether or not there are people out there that actually _want_ to do this is a different matter.

Cheers,
Trond

#7 lore raw
Quoted 4 lines · Trond Myklebust
Quoted 3 lines

Yup, same problem. Why is this allowed? Does it really work correctly?

Quoted 8 lines

Assuming that the processes have _some_ method of synchronisation, then I cannot see why it shouldn't be workable. Come to think of it, it might even be possible to use O_DIRECT to provide that synchronisation (use O_DIRECT to set a "lock" on the page, then modify it using mmap).

Whether or not there are people out there that actually _want_ to do this is a different matter.

Mixing O_DIRECT i/o and cached i/o is probably a recipe for disaster, unless the cooperating programs are very careful and very aware of how the particular file system in the particular kernel implements direct i/o and caching, including cache validation.

This seems like a dangerous enough area that denying mmap on a file which has been opened with O_DIRECT by any process and denying open(O_DIRECT) on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.

Thanx...

ps

#8 lore raw
Quoted 4 lines · Peter Staubach

This seems like a dangerous enough area that denying mmap on a file which has been opened with O_DIRECT by any process and denying open(O_DIRECT) on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.

That would be a recipe for DOSes as it would allow one process to block another just by opening a file. I'd really not like to see my database crash just because some obscure monitoring program happens to use mmap() to browse the file.

Cheers,
Trond

#9 lore raw
Quoted 11 lines · Trond Myklebust
Quoted 6 lines · Peter Staubach

This seems like a dangerous enough area that denying mmap on a file which has been opened with O_DIRECT by any process and denying open(O_DIRECT) on a file which has been mmap'd would be a good thing. These things are easy enough to keep track of, so it shouldn't be too hard to implement.

Quoted 5 lines

That would be a recipe for DOSes as it would allow one process to block another just by opening a file. I'd really not like to see my database crash just because some obscure monitoring program happens to use mmap() to browse the file.

I wouldn't think that a database would be a problem since it is opened early and then stays open. However, point granted. There are tools around, lsof and such, which would be handy for diagnosing such a situation though.

I do know of other operating systems which do implement the semantics that I have suggested and I don't think that they are concerned or have been notified that these semantics for a problem. These semantics are used because the kernel itself can not even guarantee that the data that it is caching is valid, without lots of synchronization which may tend to reduce performance.

Thanx...

ps

#10 lore raw
Quoted 7 lines · Peter Staubach

I do know of other operating systems which do implement the semantics that I have suggested and I don't think that they are concerned or have been notified
that these semantics for a problem. These semantics are used because the kernel itself can not even guarantee that the data that it is caching is valid,
without lots of synchronization which may tend to reduce performance.

I agree. It is clearly going to be a drag on performance, but it should be very much a corner case, so...

To tell you the truth, I'm more interested in this case in order to figure out how to make mmap() work correctly for the case of an ordinary file, but where the file changes on the server. Currently we just call invalidate_inode_pages2(), without unmapping first. The conclusion from Kenny's testcase appears to be that this may lead to mmapped dirty data being lost.

Cheers,
Trond

#11 lore raw
Quoted 6 lines · Trond Myklebust

I agree. It is clearly going to be a drag on performance, but it should be very much a corner case, so...

Yes, file systems tend to mostly be about the corner cases though... :-)

Quoted 7 lines

To tell you the truth, I'm more interested in this case in order to figure out how to make mmap() work correctly for the case of an ordinary file, but where the file changes on the server. Currently we just call invalidate_inode_pages2(), without unmapping first. The conclusion from Kenny's testcase appears to be that this may lead to mmapped dirty data being lost.

Ugly, huh? The options seem to be to either write out all of the data to the server or just truncate it. I have seen the former used mostly, although this does generate the "last one there wins" scenario. This does match the usual semantics associated with normal cached data from write(2)s and should fit well with the NFSv4 callback notifying that a write delegation is being withdrawn.

Thanx...

ps