Improve IOCB_NOWAIT O_DIRECT reads
Subsystems (4)
v2 cover Improve IOCB_NOWAIT O_DIRECT reads
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.
fs/iomap/direct-io.c | 23 ++++++++++++++-------- include/linux/fs.h | 2 ++ mm/filemap.c | 47 ++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 62 insertions(+), 10 deletions(-)
Since v1:
- Simplify the filemap_range_needs_writeback() loop (Willy)
- Drop the write side (Chinner)
Signature · 1 line
Jens Axboe
PATCH v2 1/3 mm: provide filemap_range_needs_writeback() helper
For O_DIRECT reads/writes, we check if we need to issue a call to filemap_write_and_wait_range() to issue and/or wait for writeback for any page in the given range. The existing mechanism just checks for a page in the range, which is suboptimal for IOCB_NOWAIT as we'll fallback to the slow path (and needing retry) if there's just a clean page cache page in the range.
Provide filemap_range_needs_writeback() which tries a little harder to check if we actually need to issue and/or wait for writeback in the range.
--- include/linux/fs.h | 2 ++ mm/filemap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)
include/linux/fs.h +2 -0
@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)extern bool filemap_range_has_page (struct address_space *,loff_t lstart,loff_t lend); +extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);extern int filemap_write_and_wait_range (struct address_space *mapping,loff_t lstart,loff_t lend); extern int __filemap_fdatawrite_range(struct address_space *mapping,
mm/filemap.c +43 -0
49 lines not on this page · raw message
Signature · 1 line
2.30.0
PATCH v2 2/3 mm: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeFor the generic page cache read helper, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT reads. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).
--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +2 -2
@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) size =i_size_read (inode);if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, iocb->ki_pos, - iocb->ki_pos + count -1 )) +if (filemap_range_needs_writeback (mapping, iocb->ki_pos, + iocb->ki_pos + count -1 ))return -EAGAIN ; }else { retval = filemap_write_and_wait_range(mapping,
Signature · 1 line
2.30.0
PATCH v2 3/3 iomap: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeFor reads, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).
--- fs/iomap/direct-io.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
fs/iomap/direct-io.c +15 -8
@@ -458,9 +458,24 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,if (pos >= dio->i_size) goto out_free_dio; +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_needs_writeback (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + }if (iter_is_iovec (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_has_page (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + } + flags |=IOMAP_WRITE ; dio->flags |=IOMAP_DIO_WRITE ;
@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->flags |=IOMAP_DIO_WRITE_FUA ; } -if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, pos, end)) { - ret = -EAGAIN ; - goto out_free_dio; - } - flags |=IOMAP_NOWAIT ; - } - ret =filemap_write_and_wait_range (mapping, pos, end);if (ret) goto out_free_dio;
Signature · 1 line
2.30.0
Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper
In reply to Jens Axboeinclude/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2637
Quoted 2 lines
include/linux/fs.h +2
-0+extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);
no need for the extern.
Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeQuoted 15 lines · Jens Axboe
For the generic page cache read helper, use the better variant of checking for the need to call
filemap_write_and_wait_range()when doingO_DIRECTreads. This avoids falling back to the slow path forIOCB_NOWAIT, if there are no pages to wait for (or write out).Signed-off-by Jens Axboe <[email protected]>--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/filemap.c +0
-0
mm/filemap.c generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) · line 2647
Quoted 7 lines
mm/filemap.c +2
-2@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) size =i_size_read (inode);if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, iocb->ki_pos, - iocb->ki_pos + count -1 )) +if (filemap_range_needs_writeback (mapping, iocb->ki_pos, + iocb->ki_pos + count -1 ))
Please avoid the overy long line, which is trivial to do by using the normal two tab indent for the continuation.
Re: [PATCH 3/3] iomap: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeQuoted 22 lines · Jens Axboe
fs/iomap/direct-io.c +15
-0+if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_needs_writeback (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + }if (iter_is_iovec (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_has_page (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + } + flags |=IOMAP_WRITE ; dio->flags |=IOMAP_DIO_WRITE ;
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 487
Quoted 10 lines
fs/iomap/direct-io.c +0
-7@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->flags |=IOMAP_DIO_WRITE_FUA ; } -if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, pos, end)) { - ret = -EAGAIN ; - goto out_free_dio; - } - flags |=IOMAP_NOWAIT ; - }
looking at this I really hate the scheme with the potential racyness and duplicated page looksups.
Why can't we pass a nonblock flag to filemap_write_and_wait_range and invalidate_inode_pages2_range that makes them return -EAGAIN when they would block to clean this whole mess up?
Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Christoph HellwigQuoted 17 lines · Christoph Hellwig
Quoted 16 lines · Jens Axboe
For the generic page cache read helper, use the better variant of checking for the need to call
filemap_write_and_wait_range()when doingO_DIRECTreads. This avoids falling back to the slow path forIOCB_NOWAIT, if there are no pages to wait for (or write out).Signed-off-by Jens Axboe <[email protected]>--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/filemap.c +0
-0@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
Quoted 9 lines
mm/filemap.c generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) · line 2647
Quoted 6 lines
mm/filemap.c +2
-2size =i_size_read (inode);if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, iocb->ki_pos, - iocb->ki_pos + count -1 )) +if (filemap_range_needs_writeback (mapping, iocb->ki_pos, + iocb->ki_pos + count -1 ))Please avoid the overy long line, which is trivial to do by using the normal two tab indent for the continuation.
Sure, fixed.
Signature · 1 line
Jens Axboe
Re: [PATCH 3/3] iomap: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Christoph HellwigQuoted 26 lines · Christoph Hellwig
Quoted 25 lines · Jens Axboe
fs/iomap/direct-io.c +15
-0+if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_needs_writeback (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + }if (iter_is_iovec (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_has_page (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + flags |=IOMAP_NOWAIT ; + } + flags |=IOMAP_WRITE ; dio->flags |=IOMAP_DIO_WRITE ;@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->flags |=IOMAP_DIO_WRITE_FUA ; }
Quoted 10 lines
fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 487
Quoted 7 lines
fs/iomap/direct-io.c +0
-7-if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, pos, end)) { - ret = -EAGAIN ; - goto out_free_dio; - } - flags |=IOMAP_NOWAIT ; - }looking at this I really hate the scheme with the potential racyness and duplicated page looksups.
Me too
Quoted 3 lines
Why can't we pass a nonblock flag to
filemap_write_and_wait_rangeandinvalidate_inode_pages2_rangethat makes them return-EAGAINwhen they would block to clean this whole mess up?
We could, but that's a _lot_ of surgery. I'd rather live with the slight race for now instead of teaching writepages, page laundering, etc about IOCB_NOWAIT.
I do think that's a worthy long term goal, but we dio read situation is bad enough that it warrants a quicker fix.
Signature · 1 line
Jens Axboe
Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper
In reply to Christoph HellwigQuoted 4 lines · Christoph Hellwig
include/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2637
Quoted 2 lines
include/linux/fs.h +2
-0+extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);no need for the extern.
Just following the style in the file.
Signature · 1 line
Jens Axboe
Andrew Morton
maintainerQuoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't
-EAGAINIOCB_NOWAITdio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.
Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.
Please remind me what prevents pages from becoming dirty during or immediately after the filemap_range_needs_writeback() check? Perhaps filemap_range_needs_writeback() could have a comment explaining what it is that keeps its return value true after it has returned it!
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't
-EAGAINIOCB_NOWAITdio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.
Quoted 3 lines
Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.
Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...
Quoted 4 lines
Please remind me what prevents pages from becoming dirty during or immediately after the
filemap_range_needs_writeback()check? Perhapsfilemap_range_needs_writeback()could have a comment explaining what it is that keeps its return value true after it has returned it!
It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified filemap_write_and_wait_range() to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.
Signature · 1 line
Jens Axboe
Quoted 28 lines · Jens Axboe
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't
-EAGAINIOCB_NOWAITdio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.Quoted 3 lines
Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.
Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...
Quoted 4 lines
Please remind me what prevents pages from becoming dirty during or immediately after the
filemap_range_needs_writeback()check? Perhapsfilemap_range_needs_writeback()could have a comment explaining what it is that keeps its return value true after it has returned it!
Quoted 6 lines
It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified
filemap_write_and_wait_range()to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.
Jens, do you have some numbers before and after your patchset is applied?
And kindly a test "profile" for FIO :-)?
Thanks.
- Sedat -
Quoted 29 lines · Sedat Dilek
Quoted 28 lines · Jens Axboe
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't
-EAGAINIOCB_NOWAITdio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.Quoted 3 lines
Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.
Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...
Quoted 4 lines
Please remind me what prevents pages from becoming dirty during or immediately after the
filemap_range_needs_writeback()check? Perhapsfilemap_range_needs_writeback()could have a comment explaining what it is that keeps its return value true after it has returned it!
Quoted 8 lines
Quoted 6 lines
It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified
filemap_write_and_wait_range()to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.Jens, do you have some numbers before and after your patchset is applied?
I don't, the load was pretty light for the test case - it was just doing 33-34K of O_DIRECT 4k random reads in a pretty small range of the device. When you end up having page cache in that range, that means you end up punting a LOT of requests to the async worker. So it wasn't as much a performance win for this particular case, but an efficiency win. You get rid of a worker using 40% CPU, and reduce the latencies.
Quoted 1 line
And kindly a test "profile" for FIO :-)?
To reproduce this, have a small range dio rand reads and then have something else that does a few buffered reads from the same range.
Signature · 1 line
Jens Axboe