Improve IOCB_NOWAIT O_DIRECT reads
Subsystems (4)
v3 cover Improve IOCB_NOWAIT O_DIRECT reads
Hi,
For v1, see:
https://lore.kernel.org/linux-fsdevel/[email protected]/
tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.
fs/iomap/direct-io.c | 24 ++++++++++++++-------- include/linux/fs.h | 2 ++ mm/filemap.c | 47 ++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 63 insertions(+), 10 deletions(-)
Andrew, any chance you can pick this up for 5.12?
Since v2:
- Drop overly long line (hch)
- Rebase to master, iomap changed flags to iomap_flags
Signature · 1 line
Jens Axboe
PATCH v3 1/3 mm: provide filemap_range_needs_writeback() helper
For O_DIRECT reads/writes, we check if we need to issue a call to filemap_write_and_wait_range() to issue and/or wait for writeback for any page in the given range. The existing mechanism just checks for a page in the range, which is suboptimal for IOCB_NOWAIT as we'll fallback to the slow path (and needing retry) if there's just a clean page cache page in the range.
Provide filemap_range_needs_writeback() which tries a little harder to check if we actually need to issue and/or wait for writeback in the range.
--- include/linux/fs.h | 2 ++ mm/filemap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)
include/linux/fs.h +2 -0
@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)extern bool filemap_range_has_page (struct address_space *,loff_t lstart,loff_t lend); +extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);extern int filemap_write_and_wait_range (struct address_space *mapping,loff_t lstart,loff_t lend); extern int __filemap_fdatawrite_range(struct address_space *mapping,
mm/filemap.c +43 -0
49 lines not on this page · raw message
Signature · 1 line
2.30.0
Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper
include/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2640
Quoted 10 lines · Jens Axboe
+++
b/include/linux/fs.hinclude/linux/fs.h +2
-0@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)extern bool filemap_range_has_page (struct address_space *,loff_t lstart,loff_t lend); +extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);extern int filemap_write_and_wait_range (struct address_space *mapping,loff_t lstart,loff_t lend); extern int __filemap_fdatawrite_range(struct address_space *mapping,
These prototypes should all be in pagemap.h, not fs.h. I can do a followup patch if you don't want to do it as part of this set. Also we're deprecating the use of 'extern' for prototypes.
Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper
Quoted 4 lines · Matthew Wilcox
Quoted 3 lines · Jens Axboe
+++
b/include/linux/fs.h(quoted diff) +0
-0@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)
Quoted 11 lines
include/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2640
Quoted 7 lines
include/linux/fs.h +2
-0extern bool filemap_range_has_page (struct address_space *,loff_t lstart,loff_t lend); +extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);extern int filemap_write_and_wait_range (struct address_space *mapping,loff_t lstart,loff_t lend); extern int __filemap_fdatawrite_range(struct address_space *mapping,These prototypes should all be in
pagemap.h, notfs.h. I can do a followup patch if you don't want to do it as part of this set. Also we're deprecating the use of 'extern' for prototypes.
Agree on both of those, but that should probably be a separate patch for both of those. extern is just following the existing style, and fs.h is the existing location...
Quoted 1 line
Reviewed-by Matthew Wilcox (Oracle) <[email protected]>
Thanks!
Signature · 1 line
Jens Axboe
Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
For
O_DIRECTreads/writes, we check if we need to issue a call tofilemap_write_and_wait_range()to issue and/or wait for writeback for any page in the given range. The existing mechanism just checks for a page in the range, which is suboptimal forIOCB_NOWAITas we'll fallback to the slow path (and needing retry) if there's just a clean page cache page in the range.
Quoted 5 lines
Provide
filemap_range_needs_writeback()which tries a little harder to check if we actually need to issue and/or wait for writeback in the range.Signed-off-by Jens Axboe <[email protected]>
Looks good to me. Feel free to add:
Honza
Quoted 75 lines
--- include/linux/fs.h | 2 ++ mm/filemap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)include/linux/fs.h +2
-0@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)extern bool filemap_range_has_page (struct address_space *,loff_t lstart,loff_t lend); +extern bool filemap_range_needs_writeback (struct address_space *, +loff_t lstart,loff_t lend);extern int filemap_write_and_wait_range (struct address_space *mapping,loff_t lstart,loff_t lend); extern int __filemap_fdatawrite_range(struct address_space *mapping,mm/filemap.c +43
-0@@ -635,6 +635,49 @@ static bool mapping_needs_writeback(struct address_space *mapping)return mapping->nrpages; } +/** +* filemap_range_needs_writeback - check if range potentially needs writeback +* @mapping: address space within which to check +* @start_byte: offset in bytes where the range starts +* @end_byte: offset in bytes where the range ends (inclusive) +* +* Find at least one page in the range supplied, usually used to check if +* direct writing in this range will trigger a writeback. Used by O_DIRECT +* read/write with IOCB_NOWAIT, to see if the caller needs to do +* filemap_write_and_wait_range() before proceeding. +* +* Return: %true if the caller should do filemap_write_and_wait_range() before +* doing O_DIRECT to a page in this range, %false otherwise. +*/ +bool filemap_range_needs_writeback (struct address_space *mapping, +loff_t start_byte,loff_t end_byte) +{ +XA_STATE (xas, &mapping->i_pages, start_byte >>PAGE_SHIFT ); +pgoff_t max = end_byte >>PAGE_SHIFT ; +struct page *page; + +if (!mapping_needs_writeback (mapping)) +return false; +if (!mapping_tagged (mapping,PAGECACHE_TAG_DIRTY ) && + !mapping_tagged (mapping,PAGECACHE_TAG_WRITEBACK )) +return false; +if (end_byte < start_byte) +return false; + +rcu_read_lock (); +xas_for_each (&xas, page, max) { +if (xas_retry (&xas, page)) +continue ; +if (xa_is_value (page)) +continue ; +if (PageDirty (page) ||PageLocked (page) ||PageWriteback (page)) +break ; + } +rcu_read_unlock (); +return page !=NULL ; +} +EXPORT_SYMBOL_GPL (filemap_range_needs_writeback); +/** * filemap_write_and_wait_range - write out & wait on a file range * @mapping: the address_space for the pages Signature · 1 line
2.30.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v3 2/3 mm: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeFor the generic page cache read helper, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT reads. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).
--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +2 -2
@@ -2645,8 +2645,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) size =i_size_read (inode);if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, iocb->ki_pos, - iocb->ki_pos + count -1 )) +if (filemap_range_needs_writeback (mapping, iocb->ki_pos, + iocb->ki_pos + count -1 ))return -EAGAIN ; }else { retval = filemap_write_and_wait_range(mapping,
Signature · 1 line
2.30.0
Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads
Quoted 6 lines · Jens Axboe
For the generic page cache read helper, use the better variant of checking for the need to call
filemap_write_and_wait_range()when doingO_DIRECTreads. This avoids falling back to the slow path forIOCB_NOWAIT, if there are no pages to wait for (or write out).Signed-off-by Jens Axboe <[email protected]>
Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeQuoted 9 lines · Jens Axboe
For the generic page cache read helper, use the better variant of checking for the need to call
filemap_write_and_wait_range()when doingO_DIRECTreads. This avoids falling back to the slow path forIOCB_NOWAIT, if there are no pages to wait for (or write out).Signed-off-by Jens Axboe <[email protected]>--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Looks good to me. Feel free to add:
Honza
Quoted 19 lines
mm/filemap.c +2
-2@@ -2645,8 +2645,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) size =i_size_read (inode);if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, iocb->ki_pos, - iocb->ki_pos + count -1 )) +if (filemap_range_needs_writeback (mapping, iocb->ki_pos, + iocb->ki_pos + count -1 ))return -EAGAIN ; }else { retval = filemap_write_and_wait_range(mapping,Signature · 1 line
2.30.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v3 3/3 iomap: use filemap_range_needs_writeback() for O_DIRECT reads
In reply to Jens AxboeFor reads, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).
--- fs/iomap/direct-io.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
fs/iomap/direct-io.c +16 -8
@@ -487,12 +487,28 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,if (pos >= dio->i_size) goto out_free_dio; +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_needs_writeback (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + iomap_flags |=IOMAP_NOWAIT ; + } +if (iter_is_iovec (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { iomap_flags |=IOMAP_WRITE ; dio->flags |=IOMAP_DIO_WRITE ; +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_has_page (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + iomap_flags |=IOMAP_NOWAIT ; + } +/* for data sync or sync, we need sync completion processing */ if (iocb->ki_flags &IOCB_DSYNC ) dio->flags |=IOMAP_DIO_NEED_SYNC ;
@@ -507,14 +523,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->flags |=IOMAP_DIO_WRITE_FUA ; } -if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, pos, end)) { - ret = -EAGAIN ; - goto out_free_dio; - } - iomap_flags |=IOMAP_NOWAIT ; - } -if (dio_flags &IOMAP_DIO_OVERWRITE_ONLY ) { ret = -EAGAIN ;if (pos >= dio->i_size || pos + count > dio->i_size)
Signature · 1 line
2.30.0
Re: [PATCH 3/3] iomap: use filemap_range_needs_writeback() for O_DIRECT reads
Quoted 6 lines · Jens Axboe
For reads, use the better variant of checking for the need to call
filemap_write_and_wait_range()when doingO_DIRECT. This avoids falling back to the slow path forIOCB_NOWAIT, if there are no pages to wait for (or write out).Signed-off-by Jens Axboe <[email protected]>
Looks good to me. Feel free to add:
Honza
Quoted 55 lines
--- fs/iomap/direct-io.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)fs/iomap/direct-io.c +16
-8@@ -487,12 +487,28 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,if (pos >= dio->i_size) goto out_free_dio; +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_needs_writeback (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + iomap_flags |=IOMAP_NOWAIT ; + } +if (iter_is_iovec (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { iomap_flags |=IOMAP_WRITE ; dio->flags |=IOMAP_DIO_WRITE ; +if (iocb->ki_flags &IOCB_NOWAIT ) { +if (filemap_range_has_page (mapping, pos, end)) { + ret = -EAGAIN ; + goto out_free_dio; + } + iomap_flags |=IOMAP_NOWAIT ; + } +/* for data sync or sync, we need sync completion processing */ if (iocb->ki_flags &IOCB_DSYNC ) dio->flags |=IOMAP_DIO_NEED_SYNC ;@@ -507,14 +523,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->flags |=IOMAP_DIO_WRITE_FUA ; } -if (iocb->ki_flags &IOCB_NOWAIT ) { -if (filemap_range_has_page (mapping, pos, end)) { - ret = -EAGAIN ; - goto out_free_dio; - } - iomap_flags |=IOMAP_NOWAIT ; - } -if (dio_flags &IOMAP_DIO_OVERWRITE_ONLY ) { ret = -EAGAIN ;if (pos >= dio->i_size || pos + count > dio->i_size)Signature · 1 line
2.30.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR