Skip to contentSkip to conversation

mm: direct io less aggressive syncs and invalidates

RFC PATCH In review Nick Piggin 5 participants 12 messages Oct 28, 2008 – Oct 30, 2008

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 mm: direct io less aggressive syncs and invalidates

Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).

Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.

This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.

Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(

---

mm/filemap.c +4 -14
Index: linux-2.6/mm/filemap.c
===================================================================
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}
 			if (retval > 0)
 				*ppos = pos + retval;
 			if (retval) {
@@ -2110,18 +2107,10 @@ generic_file_direct_write(struct kiocb *
 	if (count != ocount)
 		*nr_segs = iov_shorten((struct iovec *)iov, *nr_segs, count);
 
-	/*
-	 * Unmap all mmappings of the file up-front.
-	 *
-	 * This will cause any pte dirty bits to be propagated into the
-	 * pageframes for the subsequent filemap_write_and_wait().
-	 */
 	write_len = iov_length(iov, *nr_segs);
 	end = (pos + write_len - 1) >> PAGE_CACHE_SHIFT;
-	if (mapping_mapped(mapping))
-		unmap_mapping_range(mapping, pos, write_len, 0);
 
-	written = filemap_write_and_wait(mapping);
+	written = filemap_write_and_wait_range(mapping, pos, pos + write_len - 1);
 	if (written)
 		goto out;
 
@@ -2507,7 +2496,8 @@ generic_file_buffered_write(struct kiocb
 	 * the file data here, to try to honour O_DIRECT expectations.
 	 */
 	if (unlikely(file->f_flags & O_DIRECT) && written)
-		status = filemap_write_and_wait(mapping);
+		status = filemap_write_and_wait_range(mapping,
+					pos, pos + written - 1);
 
 	return written ? written : status;
 }
#1 lore raw
Quoted 25 lines · Nick Piggin

Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).

Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.

This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.

Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(

---
Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 9 lines
mm/filemap.c +1 -4
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

The rest looks good to me. I ran the aio-dio-regress tests against this kernel on a UP machine, and they all passed. The kernel didn't boot on my SMP box, though. Nick, any chance you could grab that test suite and run it on an smp system?
http://git.kernel.org/?p=linux/kernel/git/zab/aio-dio-regress.git;a=summary

Thanks,

Jeff

#2 lore raw
Quoted 31 lines · Jeff Moyer
Quoted 29 lines · Nick Piggin

Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).

Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.

This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.

Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(

---
Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...

Quoted 5 lines

The rest looks good to me. I ran the aio-dio-regress tests against this kernel on a UP machine, and they all passed. The kernel didn't boot on my SMP box, though. Nick, any chance you could grab that test suite and run it on an smp system?
http://git.kernel.org/?p=linux/kernel/git/zab/aio-dio-regress.git;a=summary

Yeah I could give that a shot and repost the patch for Andrew in a day or two. Thanks for looking a it.

#3 lore raw
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 4 lines

AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...

Well, that's if dio_lock_type != DIO_NO_LOCKING. cscope shows the following callers of blockdev_direct_IO_no_locking:

gfs2_direct_IO
ocfs2_direct_IO
xfs_vm_direct_IO

and of course

blkdev_direct_IO

I can't say whether all of these callers are safe. They certainly don't appear to be safe to me.

Cheers,

Jeff

#4 lore raw
Quoted 28 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 4 lines

AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...

Quoted 6 lines

Well, that's if dio_lock_type != DIO_NO_LOCKING. cscope shows the following callers of blockdev_direct_IO_no_locking:

gfs2_direct_IO
ocfs2_direct_IO
xfs_vm_direct_IO

XFS does it's own flush and invalidate before calling into the generic direct I/O code, so the above patch is safe from an XFS perspective.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw
In reply to Jeff Moyer
Quoted 35 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 4 lines

AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...

Well, that's if dio_lock_type != DIO_NO_LOCKING. cscope shows the following callers of blockdev_direct_IO_no_locking:

gfs2_direct_IO
ocfs2_direct_IO
xfs_vm_direct_IO

Quoted 6 lines

and of course

blkdev_direct_IO

I can't say whether all of these callers are safe. They certainly don't appear to be safe to me.

Ah OK of course you're right. I'll need to take another look at that and probably send any improvement as another patch.

My test SMP system just started getting memory errors for some reason so I haven't been able to boot it :( Will try to resurrect it or find another before resending...

#6 lore raw
Quoted 43 lines · Nick Piggin
Quoted 35 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 4 lines

AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...

Well, that's if dio_lock_type != DIO_NO_LOCKING. cscope shows the following callers of blockdev_direct_IO_no_locking:

gfs2_direct_IO
ocfs2_direct_IO
xfs_vm_direct_IO

Quoted 6 lines

and of course

blkdev_direct_IO

I can't say whether all of these callers are safe. They certainly don't appear to be safe to me.

Quoted 6 lines

Ah OK of course you're right. I'll need to take another look at that and probably send any improvement as another patch.

My test SMP system just started getting memory errors for some reason so I haven't been able to boot it :( Will try to resurrect it or find another before resending...

OK, I got a kernel running on an smp system for testing. I modified your patch to do a filemap_write_and_wait_range in the read case. The aio-dio-regress test suite (with a few added programs to check for buffered vs. direct I/O) passed without problems. One of those programs did not work with your initial patch, since it opened the block device and mixed buffered and direct I/O.

Cheers,

Jeff

mm/filemap.c +6 -13
index ab85536..76de63e 100644
@@ -1317,11 +1317,11 @@ generic_file_aio_read(struct kiocb *iocb, const struct iovec *iov,
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
-			if (!retval) {
+			retval = filemap_write_and_wait_range(mapping, pos,
+					pos + iov_length(iov, nr_segs) - 1);
+			if (!retval)
 				retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}
 			if (retval > 0)
 				*ppos = pos + retval;
 			if (retval) {
@@ -2123,18 +2123,10 @@ generic_file_direct_write(struct kiocb *iocb, const struct iovec *iov,
 	if (count != ocount)
 		*nr_segs = iov_shorten((struct iovec *)iov, *nr_segs, count);
 
-	/*
-	 * Unmap all mmappings of the file up-front.
-	 *
-	 * This will cause any pte dirty bits to be propagated into the
-	 * pageframes for the subsequent filemap_write_and_wait().
-	 */
 	write_len = iov_length(iov, *nr_segs);
 	end = (pos + write_len - 1) >> PAGE_CACHE_SHIFT;
-	if (mapping_mapped(mapping))
-		unmap_mapping_range(mapping, pos, write_len, 0);
 
-	written = filemap_write_and_wait(mapping);
+	written = filemap_write_and_wait_range(mapping, pos, pos + write_len - 1);
 	if (written)
 		goto out;
 
@@ -2520,7 +2512,8 @@ generic_file_buffered_write(struct kiocb *iocb, const struct iovec *iov,
 	 * the file data here, to try to honour O_DIRECT expectations.
 	 */
 	if (unlikely(file->f_flags & O_DIRECT) && written)
-		status = filemap_write_and_wait(mapping);
+		status = filemap_write_and_wait_range(mapping,
+					pos, pos + written - 1);
 
 	return written ? written : status;
 }
#7 lore raw
In reply to Jeff Moyer
Quoted 9 lines · Jeff Moyer
Quoted 9 lines

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

It seems that could be easily tested in one of the test suites, by writing a page without O_DIRECT to make a dirty page, then reading the same page with O_DIRECT. Do it a few times to be sure.

Do the test suites verify O_DIRECT / page-cache coherency?

Cheers,
-- Jamie

#8 lore raw
Quoted 19 lines · Jamie Lokier
Quoted 9 lines · Jeff Moyer
Quoted 9 lines

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +0 -1
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
Quoted 8 lines

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 5 lines
mm/filemap.c +1 -3
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 3 lines

It seems that could be easily tested in one of the test suites, by writing a page without O_DIRECT to make a dirty page, then reading the same page with O_DIRECT. Do it a few times to be sure.

Sure, are you volunteering to write this? =) For completeness, it should probably do this via mmap, too.

Quoted 1 line

Do the test suites verify O_DIRECT / page-cache coherency?

aio-dio-regress has a couple of targeted tests, but nothing I would call comprehensive. I know that during the development of the direct I/O code, there were some tests kicking around for this. The small test cases I can find include:

/*
* aiodio_sparse - issue async O_DIRECT writes to holes is a file while
* concurrently reading the file and checking that the read never reads
* uninitailized data.
*/

dio_sparse -- same theme but no async I/O

aiodio_append -- this seems to fork a child which continually seeks to the end of the file and tries to read data. Meanwhile, the parent continually appends zeroed data to the file using async direct I/O. If the child ever reads anything other than zeroes, it kicks an error.

dio_truncate --
/*
* Parent creates a zero file using DIO.
* Truncates it to zero
* Create another file with '0xaa'
*/
I have no idea why it creates a different file and fills it with 0xaa. The child process continually reads the first file, ensuring that all of the returned data is zeroes.

I'll see about incorporating the above into aio-dio-regress.

Cheers,

Jeff

#9 lore raw
Quoted 21 lines · Jeff Moyer
Quoted 19 lines · Jamie Lokier
Quoted 17 lines · Jeff Moyer
Quoted 5 lines

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311

Quoted 9 lines
mm/filemap.c +1 -4
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb
 			goto out; /* skip atime */
 		size = i_size_read(inode);
 		if (pos < size) {
-			retval = filemap_write_and_wait(mapping);
-			if (!retval) {
-				retval = mapping->a_ops->direct_IO(READ, iocb,
+			retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-			}

So why is it safe to get rid of this? Can't this result in reading stale data from disk?

Quoted 6 lines
Quoted 3 lines

It seems that could be easily tested in one of the test suites, by writing a page without O_DIRECT to make a dirty page, then reading the same page with O_DIRECT. Do it a few times to be sure.

Sure, are you volunteering to write this? =) For completeness, it should probably do this via mmap, too.

Making fsx mix buffered and direct I/O is the anwer to this problem.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw
In reply to Nick Piggin
Quoted 3 lines · Nick Piggin

Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.

mmaped memory doesn't have have to by synchronized at all with read/write. It has to be only synchonized if msync or munmap is called --- that's what Posix says.

So direct i/o implementation that ignores mmap would be correct. Direct i/o implementation that is not synchronized w.r.t. nondirect i/o wouldn't.

Mikulas

#11 lore raw
Quoted 8 lines · Mikulas Patocka
Quoted 3 lines · Nick Piggin

Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.

Quoted 6 lines

mmaped memory doesn't have have to by synchronized at all with read/write. It has to be only synchonized if msync or munmap is called --- that's what Posix says.

So direct i/o implementation that ignores mmap would be correct. Direct i/o implementation that is not synchronized w.r.t. nondirect i/o wouldn't.

Right, but our implementation tries not to ignore mmap.

Our mmap is coherent with buffered IO, and buffered IO (kinda) coherent with direct IO... least surprise says direct IO should be kinda coherent with mmap ;)

At least now there is basically no extra cost to it.