mm: direct io less aggressive syncs and invalidates
Subsystems (3)
RFC PATCH 1/1 mm: direct io less aggressive syncs and invalidates
Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).
Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.
This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.
Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(
---
mm/filemap.c +4 -14
@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping); -if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }if (retval >0 ) *ppos = pos + retval;if (retval) {
@@ -2110,18 +2107,10 @@ generic_file_direct_write(struct kiocb *if (count != ocount) *nr_segs =iov_shorten ((struct iovec *)iov, *nr_segs, count); -/* -* Unmap all mmappings of the file up-front. -* -* This will cause any pte dirty bits to be propagated into the -* pageframes for the subsequent filemap_write_and_wait(). -*/ write_len =iov_length (iov, *nr_segs); end = (pos + write_len -1 ) >>PAGE_CACHE_SHIFT ; -if (mapping_mapped (mapping)) -unmap_mapping_range (mapping, pos, write_len,0 ); - written =filemap_write_and_wait (mapping); + written =filemap_write_and_wait_range (mapping, pos, pos + write_len -1 );if (written) goto out;
@@ -2507,7 +2496,8 @@ generic_file_buffered_write(struct kiocb* the file data here, to try to honour O_DIRECT expectations. */ if (unlikely (file->f_flags &O_DIRECT ) && written) - status =filemap_write_and_wait (mapping); + status =filemap_write_and_wait_range (mapping, + pos, pos + written -1 );return written ? written : status; }
Quoted 25 lines · Nick Piggin
Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).
Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.
This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.
Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(
---
Index:linux-2.6/mm/filemap.c
===================================================================
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 9 lines
mm/filemap.c +1
-4@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping); -if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }
So why is it safe to get rid of this? Can't this result in reading stale data from disk?
The rest looks good to me. I ran the aio-dio-regress tests against this kernel on a UP machine, and they all passed. The kernel didn't boot on my SMP box, though. Nick, any chance you could grab that test suite and run it on an smp system?
http://git.kernel.org/?p=linux/kernel/git/zab/aio-dio-regress.git;a=summary
Thanks,
Jeff
Quoted 31 lines · Jeff Moyer
Quoted 29 lines · Nick Piggin
Direct IO can invalidate and sync a lot of pagecache pages in the mapping. A 4K direct IO will actually try to sync and/or invalidate the pagecache of the entire file, for example (which might be many GB or TB large).
Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.
This should fix the immediate DM deadlocks when doing direct IO reads to block device with a mounted filesystem, if only by papering over the problem somewhat rather than addressing the fsync starvation cases. Not that the patch itself is a hack, but for this particular problem it is not really the correct solution IMO. But anyway, this might be more appropriate to go into stable kernels if this DM deadlock is biting users.
Yes, I still need to put more time into finishing my pagecache tag based sync solution. Sorry :(
---
Index:linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);
Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
AFAIKS, __blockdev_direct_IO is doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...
Quoted 5 lines
The rest looks good to me. I ran the aio-dio-regress tests against this kernel on a UP machine, and they all passed. The kernel didn't boot on my SMP box, though. Nick, any chance you could grab that test suite and run it on an smp system?
http://git.kernel.org/?p=linux/kernel/git/zab/aio-dio-regress.git;a=summary
Yeah I could give that a shot and repost the patch for Andrew in a day or two. Thanks for looking a it.
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 4 lines
AFAIKS,
__blockdev_direct_IOis doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...
Well, that's if dio_lock_type != DIO_NO_LOCKING. cscope shows the following callers of blockdev_direct_IO_no_locking:
gfs2_direct_IOocfs2_direct_IOxfs_vm_direct_IO
and of course
blkdev_direct_IO
I can't say whether all of these callers are safe. They certainly don't appear to be safe to me.
Cheers,
Jeff
Quoted 28 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 4 lines
AFAIKS,
__blockdev_direct_IOis doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...
Quoted 6 lines
Well, that's if
dio_lock_type!=DIO_NO_LOCKING. cscope shows the following callers ofblockdev_direct_IO_no_locking:
gfs2_direct_IOocfs2_direct_IOxfs_vm_direct_IO
XFS does it's own flush and invalidate before calling into the generic direct I/O code, so the above patch is safe from an XFS perspective.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 35 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 4 lines
AFAIKS,
__blockdev_direct_IOis doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...Well, that's if
dio_lock_type!=DIO_NO_LOCKING. cscope shows the following callers ofblockdev_direct_IO_no_locking:
gfs2_direct_IOocfs2_direct_IOxfs_vm_direct_IO
Quoted 6 lines
and of course
blkdev_direct_IOI can't say whether all of these callers are safe. They certainly don't appear to be safe to me.
Ah OK of course you're right. I'll need to take another look at that and probably send any improvement as another patch.
My test SMP system just started getting memory errors for some reason so I haven't been able to boot it :( Will try to resurrect it or find another before resending...
Quoted 43 lines · Nick Piggin
Quoted 35 lines · Jeff Moyer
Quoted 21 lines · Nick Piggin
Quoted 11 lines · Jeff Moyer
Quoted 9 lines · Nick Piggin
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 4 lines
AFAIKS,
__blockdev_direct_IOis doing the same thing for us, when it encounters a READ. I should have documented this change. This is one thing I'm not *quite* sure of there might be a path do the block device that I haven't considered, and which does not do the sync...Well, that's if
dio_lock_type!=DIO_NO_LOCKING. cscope shows the following callers ofblockdev_direct_IO_no_locking:
gfs2_direct_IOocfs2_direct_IOxfs_vm_direct_IOQuoted 6 lines
and of course
blkdev_direct_IOI can't say whether all of these callers are safe. They certainly don't appear to be safe to me.
Quoted 6 lines
Ah OK of course you're right. I'll need to take another look at that and probably send any improvement as another patch.
My test SMP system just started getting memory errors for some reason so I haven't been able to boot it :( Will try to resurrect it or find another before resending...
OK, I got a kernel running on an smp system for testing. I modified your patch to do a filemap_write_and_wait_range in the read case. The aio-dio-regress test suite (with a few added programs to check for buffered vs. direct I/O) passed without problems. One of those programs did not work with your initial patch, since it opened the block device and mixed buffered and direct I/O.
Cheers,
Jeff
mm/filemap.c +6 -13
@@ -1317,11 +1317,11 @@ generic_file_aio_read(struct kiocb *iocb, const struct iovec *iov, goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping); -if (!retval) { + retval =filemap_write_and_wait_range (mapping, pos, + pos +iov_length (iov, nr_segs) -1 ); +if (!retval) retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }if (retval >0 ) *ppos = pos + retval;if (retval) {
@@ -2123,18 +2123,10 @@ generic_file_direct_write(struct kiocb *iocb, const struct iovec *iov,if (count != ocount) *nr_segs =iov_shorten ((struct iovec *)iov, *nr_segs, count); -/* -* Unmap all mmappings of the file up-front. -* -* This will cause any pte dirty bits to be propagated into the -* pageframes for the subsequent filemap_write_and_wait(). -*/ write_len =iov_length (iov, *nr_segs); end = (pos + write_len -1 ) >>PAGE_CACHE_SHIFT ; -if (mapping_mapped (mapping)) -unmap_mapping_range (mapping, pos, write_len,0 ); - written =filemap_write_and_wait (mapping); + written =filemap_write_and_wait_range (mapping, pos, pos + write_len -1 );if (written) goto out;
@@ -2520,7 +2512,8 @@ generic_file_buffered_write(struct kiocb *iocb, const struct iovec *iov,* the file data here, to try to honour O_DIRECT expectations. */ if (unlikely (file->f_flags &O_DIRECT ) && written) - status =filemap_write_and_wait (mapping); + status =filemap_write_and_wait_range (mapping, + pos, pos + written -1 );return written ? written : status; }
Quoted 9 lines · Jeff Moyer
Quoted 9 lines
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);
Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
It seems that could be easily tested in one of the test suites, by writing a page without O_DIRECT to make a dirty page, then reading the same page with O_DIRECT. Do it a few times to be sure.
Do the test suites verify O_DIRECT / page-cache coherency?
Cheers,
-- Jamie
Quoted 19 lines · Jamie Lokier
Quoted 9 lines · Jeff Moyer
Quoted 9 lines
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c +0
-1@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping);Quoted 8 lines
mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 5 lines
mm/filemap.c +1
-3-if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 3 lines
It seems that could be easily tested in one of the test suites, by writing a page without
O_DIRECTto make a dirty page, then reading the same page withO_DIRECT. Do it a few times to be sure.
Sure, are you volunteering to write this? =) For completeness, it should probably do this via mmap, too.
Quoted 1 line
Do the test suites verify
O_DIRECT/ page-cache coherency?
aio-dio-regress has a couple of targeted tests, but nothing I would call comprehensive. I know that during the development of the direct I/O code, there were some tests kicking around for this. The small test cases I can find include:
/*
* aiodio_sparse - issue async O_DIRECT writes to holes is a file while
* concurrently reading the file and checking that the read never reads
* uninitailized data.
*/
dio_sparse -- same theme but no async I/O
aiodio_append -- this seems to fork a child which continually seeks to the end of the file and tries to read data. Meanwhile, the parent continually appends zeroed data to the file using async direct I/O. If the child ever reads anything other than zeroes, it kicks an error.
dio_truncate --
/*
* Parent creates a zero file using DIO.
* Truncates it to zero
* Create another file with '0xaa'
*/
I have no idea why it creates a different file and fills it with 0xaa. The child process continually reads the first file, ensuring that all of the returned data is zeroes.
I'll see about incorporating the above into aio-dio-regress.
Cheers,
Jeff
Quoted 21 lines · Jeff Moyer
Quoted 19 lines · Jamie Lokier
Quoted 17 lines · Jeff Moyer
Quoted 5 lines
Index:
linux-2.6/mm/filemap.c
===================================================================mm/filemap.c generic_file_aio_read(struct kiocb *iocb · line 1311
Quoted 9 lines
mm/filemap.c +1
-4@@ -1304,11 +1304,8 @@ generic_file_aio_read(struct kiocb *iocb goto out;/* skip atime */ size =i_size_read (inode);if (pos < size) { - retval =filemap_write_and_wait (mapping); -if (!retval) { - retval = mapping->a_ops->direct_IO (READ , iocb, + retval = mapping->a_ops->direct_IO (READ , iocb, iov, pos, nr_segs); - }So why is it safe to get rid of this? Can't this result in reading stale data from disk?
Quoted 6 lines
Quoted 3 lines
It seems that could be easily tested in one of the test suites, by writing a page without
O_DIRECTto make a dirty page, then reading the same page withO_DIRECT. Do it a few times to be sure.Sure, are you volunteering to write this? =) For completeness, it should probably do this via mmap, too.
Making fsx mix buffered and direct I/O is the anwer to this problem.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 3 lines · Nick Piggin
Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.
mmaped memory doesn't have have to by synchronized at all with read/write. It has to be only synchonized if msync or munmap is called --- that's what Posix says.
So direct i/o implementation that ignores mmap would be correct. Direct i/o implementation that is not synchronized w.r.t. nondirect i/o wouldn't.
Mikulas
Quoted 8 lines · Mikulas Patocka
Quoted 3 lines · Nick Piggin
Improve this by doing range syncs. Also, memory no longer has to be unmapped to catch the dirty bits for syncing, as dirty bits would remain coherent due to dirty mmap accounting.
Quoted 6 lines
mmaped memory doesn't have have to by synchronized at all with read/write. It has to be only synchonized if msync or munmap is called --- that's what Posix says.
So direct i/o implementation that ignores mmap would be correct. Direct i/o implementation that is not synchronized w.r.t. nondirect i/o wouldn't.
Right, but our implementation tries not to ignore mmap.
Our mmap is coherent with buffered IO, and buffered IO (kinda) coherent with direct IO... least surprise says direct IO should be kinda coherent with mmap ;)
At least now there is basically no extra cost to it.