Skip to contentSkip to conversation

Improve IOCB_NOWAIT O_DIRECT reads

PATCH Superseded Jens Axboe 4 participants 14 messages Feb 9, 2021 – Feb 10, 2021

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

v2 cover Improve IOCB_NOWAIT O_DIRECT reads

Hi,

For v1, see:

https://lore.kernel.org/linux-fsdevel/[email protected]/

tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.

 fs/iomap/direct-io.c | 23 ++++++++++++++--------
 include/linux/fs.h   |  2 ++
 mm/filemap.c         | 47 ++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 62 insertions(+), 10 deletions(-)

Since v1:

- Simplify the filemap_range_needs_writeback() loop (Willy)
- Drop the write side (Chinner)

Signature · 1 line
Jens Axboe
author
#1 lore raw

PATCH v2 1/3 mm: provide filemap_range_needs_writeback() helper

For O_DIRECT reads/writes, we check if we need to issue a call to filemap_write_and_wait_range() to issue and/or wait for writeback for any page in the given range. The existing mechanism just checks for a page in the range, which is suboptimal for IOCB_NOWAIT as we'll fallback to the slow path (and needing retry) if there's just a clean page cache page in the range.

Provide filemap_range_needs_writeback() which tries a little harder to check if we actually need to issue and/or wait for writeback in the range.

---
 include/linux/fs.h |  2 ++
 mm/filemap.c       | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/fs.h +2 -0
index fd47deea7c17..def89222dfe9 100644
@@ -2633,6 +2633,8 @@ static inline int filemap_fdatawait(struct address_space *mapping)
 
 extern bool filemap_range_has_page(struct address_space *, loff_t lstart,
 				  loff_t lend);
+extern bool filemap_range_needs_writeback(struct address_space *,
+					  loff_t lstart, loff_t lend);
 extern int filemap_write_and_wait_range(struct address_space *mapping,
 				        loff_t lstart, loff_t lend);
 extern int __filemap_fdatawrite_range(struct address_space *mapping,
mm/filemap.c +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.30.0
author
#4 lore raw

PATCH v2 2/3 mm: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Jens Axboe

For the generic page cache read helper, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT reads. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).

---
 mm/filemap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +2 -2
index 6a58d50fbd31..c80acb80e8f7 100644
@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 		size = i_size_read(inode);
 		if (iocb->ki_flags & IOCB_NOWAIT) {
-			if (filemap_range_has_page(mapping, iocb->ki_pos,
-						   iocb->ki_pos + count - 1))
+			if (filemap_range_needs_writeback(mapping, iocb->ki_pos,
+							  iocb->ki_pos + count - 1))
 				return -EAGAIN;
 		} else {
 			retval = filemap_write_and_wait_range(mapping,
Signature · 1 line
2.30.0
author
#7 lore raw

PATCH v2 3/3 iomap: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Jens Axboe

For reads, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).

---
 fs/iomap/direct-io.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
fs/iomap/direct-io.c +15 -8
index 933f234d5bec..5b111d1635ab 100644
@@ -458,9 +458,24 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		if (pos >= dio->i_size)
 			goto out_free_dio;
 
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_needs_writeback(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
 		if (iter_is_iovec(iter))
 			dio->flags |= IOMAP_DIO_DIRTY;
 	} else {
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_has_page(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
+
 		flags |= IOMAP_WRITE;
 		dio->flags |= IOMAP_DIO_WRITE;
 
@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			dio->flags |= IOMAP_DIO_WRITE_FUA;
 	}
 
-	if (iocb->ki_flags & IOCB_NOWAIT) {
-		if (filemap_range_has_page(mapping, pos, end)) {
-			ret = -EAGAIN;
-			goto out_free_dio;
-		}
-		flags |= IOMAP_NOWAIT;
-	}
-
 	ret = filemap_write_and_wait_range(mapping, pos, end);
 	if (ret)
 		goto out_free_dio;
Signature · 1 line
2.30.0
#2 lore raw

Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper

In reply to Jens Axboe

include/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2637

Quoted 2 lines
include/linux/fs.h +2 -0
+extern bool filemap_range_needs_writeback(struct address_space *,
+					  loff_t lstart, loff_t lend);

no need for the extern.

#5 lore raw

Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Jens Axboe
Quoted 15 lines · Jens Axboe

For the generic page cache read helper, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT reads. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).

---
 mm/filemap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +0 -0
index 6a58d50fbd31..c80acb80e8f7 100644

mm/filemap.c generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) · line 2647

Quoted 7 lines
mm/filemap.c +2 -2
@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 		size = i_size_read(inode);
 		if (iocb->ki_flags & IOCB_NOWAIT) {
-			if (filemap_range_has_page(mapping, iocb->ki_pos,
-						   iocb->ki_pos + count - 1))
+			if (filemap_range_needs_writeback(mapping, iocb->ki_pos,
+							  iocb->ki_pos + count - 1))

Please avoid the overy long line, which is trivial to do by using the normal two tab indent for the continuation.

#8 lore raw

Re: [PATCH 3/3] iomap: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Jens Axboe
Quoted 22 lines · Jens Axboe
fs/iomap/direct-io.c +15 -0
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_needs_writeback(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
 		if (iter_is_iovec(iter))
 			dio->flags |= IOMAP_DIO_DIRTY;
 	} else {
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_has_page(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
+
 		flags |= IOMAP_WRITE;
 		dio->flags |= IOMAP_DIO_WRITE;

fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 487

Quoted 10 lines
fs/iomap/direct-io.c +0 -7
@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			dio->flags |= IOMAP_DIO_WRITE_FUA;
 	}
 
-	if (iocb->ki_flags & IOCB_NOWAIT) {
-		if (filemap_range_has_page(mapping, pos, end)) {
-			ret = -EAGAIN;
-			goto out_free_dio;
-		}
-		flags |= IOMAP_NOWAIT;
-	}

looking at this I really hate the scheme with the potential racyness and duplicated page looksups.

Why can't we pass a nonblock flag to filemap_write_and_wait_range and invalidate_inode_pages2_range that makes them return -EAGAIN when they would block to clean this whole mess up?

author
#6 lore raw

Re: [PATCH 2/3] mm: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Christoph Hellwig
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Jens Axboe

For the generic page cache read helper, use the better variant of checking for the need to call filemap_write_and_wait_range() when doing O_DIRECT reads. This avoids falling back to the slow path for IOCB_NOWAIT, if there are no pages to wait for (or write out).

---
 mm/filemap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +0 -0
index 6a58d50fbd31..c80acb80e8f7 100644
@@ -2643,8 +2643,8 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
Quoted 9 lines

mm/filemap.c generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) · line 2647

Quoted 6 lines
mm/filemap.c +2 -2
 		size = i_size_read(inode);
 		if (iocb->ki_flags & IOCB_NOWAIT) {
-			if (filemap_range_has_page(mapping, iocb->ki_pos,
-						   iocb->ki_pos + count - 1))
+			if (filemap_range_needs_writeback(mapping, iocb->ki_pos,
+							  iocb->ki_pos + count - 1))

Please avoid the overy long line, which is trivial to do by using the normal two tab indent for the continuation.

Sure, fixed.

Signature · 1 line
Jens Axboe
author
#9 lore raw

Re: [PATCH 3/3] iomap: use filemap_range_needs_writeback() for O_DIRECT reads

In reply to Christoph Hellwig
Quoted 26 lines · Christoph Hellwig
Quoted 25 lines · Jens Axboe
fs/iomap/direct-io.c +15 -0
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_needs_writeback(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
 		if (iter_is_iovec(iter))
 			dio->flags |= IOMAP_DIO_DIRTY;
 	} else {
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (filemap_range_has_page(mapping, pos, end)) {
+				ret = -EAGAIN;
+				goto out_free_dio;
+			}
+			flags |= IOMAP_NOWAIT;
+		}
+
 		flags |= IOMAP_WRITE;
 		dio->flags |= IOMAP_DIO_WRITE;
@@ -478,14 +493,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			dio->flags |= IOMAP_DIO_WRITE_FUA;
 	}
 
Quoted 10 lines

fs/iomap/direct-io.c __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, · line 487

Quoted 7 lines
fs/iomap/direct-io.c +0 -7
-	if (iocb->ki_flags & IOCB_NOWAIT) {
-		if (filemap_range_has_page(mapping, pos, end)) {
-			ret = -EAGAIN;
-			goto out_free_dio;
-		}
-		flags |= IOMAP_NOWAIT;
-	}

looking at this I really hate the scheme with the potential racyness and duplicated page looksups.

Me too

Quoted 3 lines

Why can't we pass a nonblock flag to filemap_write_and_wait_range and invalidate_inode_pages2_range that makes them return -EAGAIN when they would block to clean this whole mess up?

We could, but that's a _lot_ of surgery. I'd rather live with the slight race for now instead of teaching writepages, page laundering, etc about IOCB_NOWAIT.

I do think that's a worthy long term goal, but we dio read situation is bad enough that it warrants a quicker fix.

Signature · 1 line
Jens Axboe
author
#3 lore raw

Re: [PATCH 1/3] mm: provide filemap_range_needs_writeback() helper

In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

include/linux/fs.h static inline int filemap_fdatawait(struct address_space *mapping) · line 2637

Quoted 2 lines
include/linux/fs.h +2 -0
+extern bool filemap_range_needs_writeback(struct address_space *,
+					  loff_t lstart, loff_t lend);

no need for the extern.

Just following the style in the file.

Signature · 1 line
Jens Axboe
maintainer
#10 lore raw
In reply to Jens Axboe
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:

Hi,

For v1, see:

https://lore.kernel.org/linux-fsdevel/[email protected]/

tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.

Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.

Please remind me what prevents pages from becoming dirty during or immediately after the filemap_range_needs_writeback() check? Perhaps filemap_range_needs_writeback() could have a comment explaining what it is that keeps its return value true after it has returned it!

author
#11 lore raw
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:

Hi,

For v1, see:

https://lore.kernel.org/linux-fsdevel/[email protected]/

tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.

Quoted 3 lines

Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.

Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...

Quoted 4 lines

Please remind me what prevents pages from becoming dirty during or immediately after the filemap_range_needs_writeback() check? Perhaps filemap_range_needs_writeback() could have a comment explaining what it is that keeps its return value true after it has returned it!

It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified filemap_write_and_wait_range() to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.

Signature · 1 line
Jens Axboe
#12 lore raw
Quoted 28 lines · Jens Axboe
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:

Hi,

For v1, see:

https://lore.kernel.org/linux-fsdevel/[email protected]/

tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.

Quoted 3 lines

Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.

Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...

Quoted 4 lines

Please remind me what prevents pages from becoming dirty during or immediately after the filemap_range_needs_writeback() check? Perhaps filemap_range_needs_writeback() could have a comment explaining what it is that keeps its return value true after it has returned it!

Quoted 6 lines

It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified filemap_write_and_wait_range() to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.

Jens, do you have some numbers before and after your patchset is applied?

And kindly a test "profile" for FIO :-)?

Thanks.

- Sedat -

author
#13 lore raw
Quoted 29 lines · Sedat Dilek
Quoted 28 lines · Jens Axboe
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Feb 2021 19:30:05 -0700 Jens Axboe <[email protected]> wrote:

Hi,

For v1, see:

https://lore.kernel.org/linux-fsdevel/[email protected]/

tldr; don't -EAGAIN IOCB_NOWAIT dio reads just because we have page cache entries for the given range. This causes unnecessary work from the callers side, when the IO could have been issued totally fine without blocking on writeback when there is none.

Quoted 3 lines

Seems a good idea. Obviously we'll do more work in the case where some writeback needs doing, but we'll be doing synchronous writeout in that case anyway so who cares.

Right, I think that'll be a round two on top of this, so we can make the write side happier too. That's a bit more involved...

Quoted 4 lines

Please remind me what prevents pages from becoming dirty during or immediately after the filemap_range_needs_writeback() check? Perhaps filemap_range_needs_writeback() could have a comment explaining what it is that keeps its return value true after it has returned it!

Quoted 8 lines
Quoted 6 lines

It's inherently racy, just like it is now. There's really no difference there, and I don't think there's a way to close that. Even if you modified filemap_write_and_wait_range() to be non-block friendly, there's nothing stopping anyone from adding dirty page cache right after that call.

Jens, do you have some numbers before and after your patchset is applied?

I don't, the load was pretty light for the test case - it was just doing 33-34K of O_DIRECT 4k random reads in a pretty small range of the device. When you end up having page cache in that range, that means you end up punting a LOT of requests to the async worker. So it wasn't as much a performance win for this particular case, but an efficiency win. You get rid of a worker using 40% CPU, and reduce the latencies.

Quoted 1 line

And kindly a test "profile" for FIO :-)?

To reproduce this, have a small range dio rand reads and then have something else that does a few buffered reads from the same range.

Signature · 1 line
Jens Axboe