btrfs: some speedups around nowait dio
cover btrfs: some speedups around nowait dio
From: Filipe Manana <[email protected]>
This patchset makes our direct IO code behave better for NOWAIT writes, avoiding blocking in several places for potentially long periods due to waits for IO. It also removes running the same nocow checks twice (which can be expensive) and doing extra path allocations. The last patch in the series has a test and the results I got before and after applying this patchset.
Filipe Manana (8):
btrfs: avoid blocking on page locks with nowait dio on compressed range
btrfs: avoid blocking nowait dio when locking file range
btrfs: avoid double nocow check when doing nowait dio writes
btrfs: stop allocating a path when checking if cross reference exists
btrfs: free path at can_nocow_extent() before checking for checksum items
btrfs: release path earlier at can_nocow_extent()
btrfs: avoid blocking when allocating context for nowait dio read/write
btrfs: avoid blocking on space revervation when doing nowait dio writes
fs/btrfs/ctree.h | 5 +- fs/btrfs/delalloc-space.c | 9 +-- fs/btrfs/extent-tree.c | 9 +-- fs/btrfs/file.c | 104 +++++++++++----------------------- fs/btrfs/inode.c | 116 ++++++++++++++++++++++++++++---------- fs/btrfs/qgroup.c | 5 +- fs/btrfs/qgroup.h | 12 ++-- fs/btrfs/relocation.c | 3 +- fs/btrfs/root-tree.c | 3 +- 9 files changed, 144 insertions(+), 122 deletions(-)
Signature · 1 line
2.33.0
PATCH 1/8 btrfs: avoid blocking on page locks with nowait dio on compressed range
From: Filipe Manana <[email protected]>
If we are doing NOWAIT direct IO read/write and our inode has compressed extents, we call filemap_fdatawrite_range() against the range in order to wait for compressed writeback to complete, since the generic code at iomap_dio_rw() calls filemap_write_and_wait_range() once, which is not enough to wait for compressed writeback to complete.
This call to filemap_fdatawrite_range() can block on page locks, since the first writepages() on a range that we will try to compress results only in queuing a work to compress the data while holding the pages locked.
Even though the generic code at iomap_dio_rw() will do the right thing and return -EAGAIN for NOWAIT requests in case there are pages in the range, we can still end up at btrfs_dio_iomap_begin() with pages in the range because either of the following can happen:
1) Memory mapped writes, as we haven't locked the range yet;
2) Buffered reads might have started, which lock the pages, and we do the filemap_fdatawrite_range() call before locking the file range.
So don't call filemap_fdatawrite_range() at btrfs_dio_iomap_begin() if we are doing a NOWAIT read/write. Instead call filemap_range_needs_writeback() to check if there are any locked, dirty, or under writeback pages, and return -EAGAIN if that's the case.
--- fs/btrfs/inode.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-)
fs/btrfs/inode.c +26 -8
@@ -7537,17 +7537,35 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, lockend = start + len -1 ;/* -* The generic stuff only does filemap_write_and_wait_range, which -* isn't enough if we've written compressed pages to this area, so we -* need to flush the dirty pages again to make absolutely sure that any -* outstanding dirty pages are on disk. +* iomap_dio_rw() only does filemap_write_and_wait_range(), which isn't +* enough if we've written compressed pages to this area, so we need to +* flush the dirty pages again to make absolutely sure that any +* outstanding dirty pages are on disk - the first flush only starts +* compression on the data, while keeping the pages locked, so by the +* time the second flush returns we know bios for the compressed pages +* were submitted and finished, and the pages no longer under writeback. +* +* If we have a NOWAIT request and we have any pages in the range that +* are locked, likely due to compression still in progress, we don't want +* to block on page locks. We also don't want to block on pages marked as +* dirty or under writeback (same as for the non-compression case). +* iomap_dio_rw() did the same check, but after that and before we got +* here, mmap'ed writes may have happened or buffered reads started +* (readpage() and readahead(), which lock pages), as we haven't locked +* the file range yet. */ if (test_bit (BTRFS_INODE_HAS_ASYNC_EXTENT , &BTRFS_I (inode)->runtime_flags)) { - ret =filemap_fdatawrite_range (inode->i_mapping, start, - start + length -1 ); -if (ret) -return ret; +if (flags &IOMAP_NOWAIT ) { +if (filemap_range_needs_writeback (inode->i_mapping, + lockstart, lockend)) +return -EAGAIN ; + }else { + ret =filemap_fdatawrite_range (inode->i_mapping, start, + start + length -1 ); +if (ret) +return ret; + } } dio_data =kzalloc (sizeof (*dio_data),GFP_NOFS );
Signature · 1 line
2.33.0
PATCH 2/8 btrfs: avoid blocking nowait dio when locking file range
In reply to [email protected]From: Filipe Manana <[email protected]>
If we are doing a NOWAIT direct IO read/write, we can block when locking the file range at btrfs_dio_iomap_begin(), as it's possible the range (or a part of it) is already locked by another task (mmap writes, another direct IO read/write racing with us, fiemap, etc). We are also waiting for completion of any ordered extent we find in the range, which also can block us for a significant amount of time.
There's also the incorrect fallback to buffered IO (returning -ENOTBLK) when we are dealing with a NOWAIT request and we can't proceed. In this case we should be returning -EAGAIN, as falling back to buffered IO can result in blocking for many different reasons, so that the caller can delegate a retry to a context where blocking is more acceptable.
Fix these cases by:
1) Doing a try lock on the file range and failing with -EAGAIN if we can not lock right away;
2) Fail with -EAGAIN if we find an ordered extent;
3) Return -EAGAIN instead of -ENOTBLK when we need to fallback to buffered IO and we have a NOWAIT request.
This will also allow us to avoid a duplicated check that verifies if we are able to do a NOCOW write for NOWAIT direct IO writes, done in the next patch.
--- fs/btrfs/inode.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-)
fs/btrfs/inode.c +23 -11
@@ -7255,14 +7255,22 @@ noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len, }static int lock_extent_direct (struct inode *inode,u64 lockstart,u64 lockend, -struct extent_state **cached_state,bool writing) +struct extent_state **cached_state, +unsigned int iomap_flags) { +const bool writing = (iomap_flags &IOMAP_WRITE ); +const bool nowait = (iomap_flags &IOMAP_NOWAIT ); +struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct btrfs_ordered_extent *ordered;int ret =0 ;while (1 ) { -lock_extent_bits (&BTRFS_I (inode)->io_tree, lockstart, lockend, - cached_state); +if (nowait) { +if (!try_lock_extent (io_tree, lockstart, lockend)) +return -EAGAIN ; + }else { +lock_extent_bits (io_tree, lockstart, lockend, cached_state); + }/* * We're concerned with the entire range that we're going to be * doing DIO to, so we need to make sure there's no ordered
@@ -7283,10 +7291,14 @@ static int lock_extent_direct(struct inode *inode, u64 lockstart, u64 lockend, lockstart, lockend)))break ; -unlock_extent_cached (&BTRFS_I (inode)->io_tree, lockstart, lockend, - cached_state); +unlock_extent_cached (io_tree, lockstart, lockend, cached_state);if (ordered) { +if (nowait) { +btrfs_put_ordered_extent (ordered); + ret = -EAGAIN ; +break ; + }/* * If we are doing a DIO read and the ordered extent we * found is for a buffered write, we can not wait for it
@@ -7306,7 +7318,7 @@ static int lock_extent_direct(struct inode *inode, u64 lockstart, u64 lockend, test_bit(BTRFS_ORDERED_DIRECT, &ordered->flags))btrfs_start_ordered_extent (ordered,1 ); else - ret = -ENOTBLK ; + ret = nowait ? -EAGAIN : -ENOTBLK ;btrfs_put_ordered_extent (ordered); }else {/*
@@ -7322,7 +7334,7 @@ static int lock_extent_direct(struct inode *inode, u64 lockstart, u64 lockend,* ordered extent to complete while holding a lock on * that page. */ - ret = -ENOTBLK ; + ret = nowait ? -EAGAIN : -ENOTBLK ; }if (ret)
@@ -7577,12 +7589,12 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start,/* * If this errors out it's because we couldn't invalidate pagecache for -* this range and we need to fallback to buffered. +* this range and we need to fallback to buffered IO, or we are doing a +* NOWAIT read/write and we need to block. */ -if (lock_extent_direct (inode, lockstart, lockend, &cached_state, write)) { - ret = -ENOTBLK ; + ret =lock_extent_direct (inode, lockstart, lockend, &cached_state, flags); +if (ret <0 ) goto err; - } em =btrfs_get_extent (BTRFS_I (inode),NULL ,0 , start, len);if (IS_ERR (em)) {
Signature · 1 line
2.33.0
PATCH 3/8 btrfs: avoid double nocow check when doing nowait dio writes
In reply to [email protected]From: Filipe Manana <[email protected]>
When doing a NOWAIT direct IO write we are checking twice if we can COW into the target file range using can_nocow_extent() - once at the very beginning of the write path, at btrfs_write_check() via check_nocow_nolock(), and later again at btrfs_get_blocks_direct_write().
The can_nocow_extent() function does a lot of expensive things - searching for the file extent item in the inode's subvolume tree, searching for the extent item in the extent tree, checking delayed references, etc, so it isn't a very cheap call.
We can remove the first check at btrfs_write_check(), and add there a quick check to verify if the inode has the NODATACOW or PREALLOC flags, and quickly bail out if it doesn't have neither of those flags, as that means we have to COW and therefore can't comply with the NOWAIT semantics.
After this we do only one call to can_nocow_extent(), while we are at btrfs_get_blocks_direct_write(), where we have already locked the file range and we did a try lock on the range before, at btrfs_dio_iomap_begin() (since the previous patch in the series).
--- fs/btrfs/file.c | 102 +++++++++++++++-------------------------------- fs/btrfs/inode.c | 8 +++- 2 files changed, 39 insertions(+), 71 deletions(-)
fs/btrfs/file.c +33 -69
@@ -1460,8 +1460,27 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages,return ret; } -static int check_can_nocow (struct btrfs_inode *inode,loff_t pos, -size_t *write_bytes,bool nowait) +/* +* Check if we can do nocow write into the range [@pos, @pos + @write_bytes) +* +* @pos: File offset. +* @write_bytes: The length to write, will be updated to the nocow writeable +* range. +* +* This function will flush ordered extents in the range to ensure proper +* nocow checks. +* +* Return: +* > 0 If we can nocow, and updates @write_bytes. +* 0 If we can't do a nocow write. +* -EAGAIN If we can't do a nocow write because snapshoting of the inode's +* root is in progress. +* < 0 If an error happened. +* +* NOTE: Callers need to call btrfs_check_nocow_unlock() if we return > 0. +*/ +int btrfs_check_nocow_lock (struct btrfs_inode *inode,loff_t pos, +size_t *write_bytes) {struct btrfs_fs_info *fs_info = inode->root->fs_info;struct btrfs_root *root = inode->root;
@@ -1472,7 +1491,7 @@ static int check_can_nocow(struct btrfs_inode *inode, loff_t pos,if (!(inode->flags & (BTRFS_INODE_NODATACOW |BTRFS_INODE_PREALLOC )))return 0 ; -if (!nowait && !btrfs_drew_try_write_lock (&root->snapshot_lock)) +if (!btrfs_drew_try_write_lock (&root->snapshot_lock))return -EAGAIN ; lockstart =round_down (pos, fs_info->sectorsize);
@@ -1480,71 +1499,21 @@ static int check_can_nocow(struct btrfs_inode *inode, loff_t pos, fs_info->sectorsize) - 1; num_bytes = lockend - lockstart +1 ; -if (nowait) { -struct btrfs_ordered_extent *ordered; - -if (!try_lock_extent (&inode->io_tree, lockstart, lockend)) -return -EAGAIN ; - - ordered =btrfs_lookup_ordered_range (inode, lockstart, - num_bytes); -if (ordered) { -btrfs_put_ordered_extent (ordered); - ret = -EAGAIN ; - goto out_unlock; - } - }else { -btrfs_lock_and_flush_ordered_range (inode, lockstart, - lockend,NULL ); - } - +btrfs_lock_and_flush_ordered_range (inode, lockstart, lockend,NULL ); ret =can_nocow_extent (&inode->vfs_inode, lockstart, &num_bytes,NULL ,NULL ,NULL , false);if (ret <=0 ) { ret =0 ; -if (!nowait) -btrfs_drew_write_unlock (&root->snapshot_lock); +btrfs_drew_write_unlock (&root->snapshot_lock); }else { *write_bytes =min_t (size_t, *write_bytes , num_bytes - pos + lockstart); } -out_unlock:unlock_extent (&inode->io_tree, lockstart, lockend);return ret; } -static int check_nocow_nolock (struct btrfs_inode *inode,loff_t pos, -size_t *write_bytes) -{ -return check_can_nocow (inode, pos, write_bytes, true); -} - -/* -* Check if we can do nocow write into the range [@pos, @pos + @write_bytes) -* -* @pos: File offset -* @write_bytes: The length to write, will be updated to the nocow writeable -* range -* -* This function will flush ordered extents in the range to ensure proper -* nocow checks. -* -* Return: -* >0 and update @write_bytes if we can do nocow write -* 0 if we can't do nocow write -* -EAGAIN if we can't get the needed lock or there are ordered extents -* for * (nowait == true) case -* <0 if other error happened -* -* NOTE: Callers need to release the lock by btrfs_check_nocow_unlock(). -*/ -int btrfs_check_nocow_lock (struct btrfs_inode *inode,loff_t pos, -size_t *write_bytes) -{ -return check_can_nocow (inode, pos, write_bytes, false); -} -void btrfs_check_nocow_unlock (struct btrfs_inode *inode) {btrfs_drew_write_unlock (&inode->root->snapshot_lock);
@@ -1579,20 +1548,15 @@ static int btrfs_write_check(struct kiocb *iocb, struct iov_iter *from,loff_t oldsize;loff_t start_pos; -if (iocb->ki_flags &IOCB_NOWAIT ) { -size_t nocow_bytes = count; - -/* We will allocate space in case nodatacow is not set, so bail */ -if (check_nocow_nolock (BTRFS_I (inode), pos, &nocow_bytes) <=0 ) -return -EAGAIN ; -/* -* There are holes in the range or parts of the range that must -* be COWed (shared extents, RO block groups, etc), so just bail -* out. -*/ -if (nocow_bytes < count) -return -EAGAIN ; - } +/* +* Quickly bail out on NOWAIT writes if we don't have the nodatacow or +* prealloc flags, as without those flags we always have to COW. We will +* later check if we can really COW into the target range (using +* can_nocow_extent() at btrfs_get_blocks_direct_write()). +*/ +if ((iocb->ki_flags &IOCB_NOWAIT ) && + !(BTRFS_I (inode)->flags & (BTRFS_INODE_NODATACOW |BTRFS_INODE_PREALLOC ))) +return -EAGAIN ; current->backing_dev_info =inode_to_bdi (inode); ret =file_remove_privs (file);
fs/btrfs/inode.c +6 -2
@@ -7408,7 +7408,8 @@ static struct extent_map *create_io_em(struct btrfs_inode *inode, u64 start,static int btrfs_get_blocks_direct_write (struct extent_map **map,struct inode *inode,struct btrfs_dio_data *dio_data, -u64 start,u64 len) +u64 start,u64 len, +unsigned int iomap_flags) {struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_map *em = *map;
@@ -7478,6 +7479,9 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map,free_extent_map (em); *map =NULL ; +if (iomap_flags &IOMAP_NOWAIT ) +return -EAGAIN ; +/* We have to COW, so need to reserve metadata and data space. */ ret = btrfs_delalloc_reserve_space(BTRFS_I(inode), &dio_data->data_reserved,
@@ -7654,7 +7658,7 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start,if (write) { ret =btrfs_get_blocks_direct_write (&em, inode, dio_data, - start, len); + start, len, flags);if (ret <0 ) goto unlock_err; unlock_extents = true;
Signature · 1 line
2.33.0
PATCH 4/8 btrfs: stop allocating a path when checking if cross reference exists
In reply to [email protected]From: Filipe Manana <[email protected]>
At btrfs_cross_ref_exist() we always allocate a path, but we really don't need to because all its callers (only 2) already have an allocated path that is not being used when they call btrfs_cross_ref_exist(). So change btrfs_cross_ref_exist() to take a path as an argument and update both its callers to pass in the unused path they have when they call btrfs_cross_ref_exist().
--- fs/btrfs/ctree.h | 3 ++- fs/btrfs/extent-tree.c | 9 ++------- fs/btrfs/inode.c | 5 +++-- 3 files changed, 7 insertions(+), 10 deletions(-)
fs/btrfs/ctree.h +2 -1
@@ -2783,7 +2783,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans, u64 bytenr, u64 num_bytes);int btrfs_exclude_logged_extents (struct extent_buffer *eb);int btrfs_cross_ref_exist (struct btrfs_root *root, -u64 objectid,u64 offset,u64 bytenr,bool strict); +u64 objectid,u64 offset,u64 bytenr,bool strict, +struct btrfs_path *path); struct extent_buffer *btrfs_alloc_tree_block(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 parent, u64 root_objectid,
fs/btrfs/extent-tree.c +2 -7
@@ -2357,15 +2357,10 @@ static noinline int check_committed_ref(struct btrfs_root *root, }int btrfs_cross_ref_exist (struct btrfs_root *root,u64 objectid,u64 offset, -u64 bytenr,bool strict) +u64 bytenr,bool strict,struct btrfs_path *path) { -struct btrfs_path *path;int ret; - path =btrfs_alloc_path (); -if (!path) -return -ENOMEM ; - do { ret = check_committed_ref(root, path, objectid, offset, bytenr, strict);
@@ -2376,7 +2371,7 @@ int btrfs_cross_ref_exist(struct btrfs_root *root, u64 objectid, u64 offset, }while (ret == -EAGAIN ); out: -btrfs_free_path (path); +btrfs_release_path (path);if (btrfs_is_data_reloc_root (root))WARN_ON (ret >0 );return ret;
fs/btrfs/inode.c +3 -2
@@ -1800,7 +1800,8 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode, ret =btrfs_cross_ref_exist (root, ino, found_key.offset - - extent_offset, disk_bytenr, false); + extent_offset, disk_bytenr, + false, path);if (ret) {/* * ret could be -EIO if the above fails to read
@@ -7227,7 +7228,7 @@ noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len, ret =btrfs_cross_ref_exist (root,btrfs_ino (BTRFS_I (inode)), key.offset - backref_offset, disk_bytenr, - strict); + strict, path);if (ret) { ret =0 ; goto out;
Signature · 1 line
2.33.0
PATCH 5/8 btrfs: free path at can_nocow_extent() before checking for checksum items
In reply to [email protected]From: Filipe Manana <[email protected]>
When we look for checksum items, through csum_exist_in_range(), at can_nocow_extent(), we no longer need the path that we have previously allocated. Through csum_exist_in_range() -> btrfs_lookup_csums_range(), we also end up allocating a path, so we are adding unnecessary extra memory usage. So free the path before calling csum_exist_in_range().
--- fs/btrfs/inode.c | 8 ++++++++ 1 file changed, 8 insertions(+)
fs/btrfs/inode.c +8 -0
@@ -7234,6 +7234,14 @@ noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len, goto out; } +/* +* We don't need the path anymore, plus through the csum_exist_in_range() +* call below we will end up allocating another path. So free the path +* to avoid unnecessary extra memory usage. +*/ +btrfs_free_path (path); + path =NULL ; +/* * adjust disk_bytenr and num_bytes to cover just the bytes * in this extent we are about to write. If there
Signature · 1 line
2.33.0
PATCH 6/8 btrfs: release path earlier at can_nocow_extent()
In reply to [email protected]From: Filipe Manana <[email protected]>
At can_nocow_extent(), we are releasing the path only after checking if the block group that has the target extent is read only, and after checking if there's delalloc in the range in case our extent is a preallocated extent. The read only extent check can be expensive if we have a very large filesystem with many block groups, as well as the check for delalloc in the inode's io_tree in case the io_tree is big due to IO on other file ranges.
Our path is holding a read lock on a leaf and there's no need to keep the lock while doing those two checks, so release the path before doing them, immediately after the last use of the leaf.
--- fs/btrfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/btrfs/inode.c +2 -2
@@ -7202,6 +7202,8 @@ noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len, *ram_bytes =btrfs_file_extent_ram_bytes (leaf, fi); } +btrfs_release_path (path); +if (btrfs_extent_readonly (fs_info, disk_bytenr)) goto out;
@@ -7219,8 +7221,6 @@ noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len, } } -btrfs_release_path (path); -/* * look for other files referencing this extent, if we * find any we must cow
Signature · 1 line
2.33.0
PATCH 7/8 btrfs: avoid blocking when allocating context for nowait dio read/write
In reply to [email protected]From: Filipe Manana <[email protected]>
When doing a NOWAIT direct IO read/write, we allocate a context object (struct btrfs_dio_data) with GFP_NOFS, which can result in blocking waiting for memory allocation (GFP_NOFS is __GFP_RECLAIM | __GFP_IO). This is undesirable for the NOWAIT semantics, so do the allocation with GFP_NOWAIT if we are serving a NOWAIT request and if the allocation fails return -EAGAIN, so that the caller can fallback to a blocking context and retry with a non-blocking write.
--- fs/btrfs/inode.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
fs/btrfs/inode.c +9 -3
@@ -7593,9 +7593,15 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, } } - dio_data =kzalloc (sizeof (*dio_data),GFP_NOFS ); -if (!dio_data) -return -ENOMEM ; +if (flags &IOMAP_NOWAIT ) { + dio_data =kzalloc (sizeof (*dio_data),GFP_NOWAIT ); +if (!dio_data) +return -EAGAIN ; + }else { + dio_data =kzalloc (sizeof (*dio_data),GFP_NOFS ); +if (!dio_data) +return -ENOMEM ; + } iomap->private = dio_data;
Signature · 1 line
2.33.0
PATCH 8/8 btrfs: avoid blocking on space revervation when doing nowait dio writes
In reply to [email protected]From: Filipe Manana <[email protected]>
When doing a NOWAIT direct IO write, if we can NOCOW then it means we can proceed with the non-blocking, NOWAIT path. However reserving the metadata space and qgroup meta space can often result in blocking - flushing delalloc, wait for ordered extents to complete, trigger transaction commits, etc, going against the semantics of a NOWAIT write.
So make the NOWAIT write path to try to reserve all the metadata it needs without resulting in a blocking behaviour - if we get -ENOSPC or -EDQUOT then return -EAGAIN to make the caller fallback to a blocking direct IO write.
This is part of a patchset comprised of the following patches:
btrfs: avoid blocking on page locks with nowait dio on compressed range
btrfs: avoid blocking nowait dio when locking file range
btrfs: avoid double nocow check when doing nowait dio writes
btrfs: stop allocating a path when checking if cross reference exists
btrfs: free path at can_nocow_extent() before checking for checksum items
btrfs: release path earlier at can_nocow_extent()
btrfs: avoid blocking when allocating context for nowait dio read/write
btrfs: avoid blocking on space revervation when doing nowait dio writes
The following test was run before and after applying this patchset:
$ cat io-uring-nodatacow-test.sh #!/bin/bash
DEV=/dev/sdc
MNT=/mnt/sdc
MOUNT_OPTIONS="-o ssd -o nodatacow"
MKFS_OPTIONS="-R free-space-tree -O no-holes"
NUM_JOBS=4
FILE_SIZE=8G
RUN_TIME=300
cat <<EOF > /tmp/fio-job.ini
[io_uring_rw]
rw=randrw
fsync=0
fallocate=posix
group_reporting=1
direct=1
ioengine=io_uring
iodepth=64 bssplit=4k/20:8k/20:16k/20:32k/10:64k/10:128k/5:256k/5:512k/5:1m/5 filesize=$FILE_SIZE
runtime=$RUN_TIME
time_based
filename=foobar
directory=$MNT
numjobs=$NUM_JOBS
thread
EOF
echo performance | \
tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor umount $MNT &> /dev/null
mkfs.btrfs -f $MKFS_OPTIONS $DEV &> /dev/null
mount $MOUNT_OPTIONS $DEV $MNT
fio /tmp/fio-job.ini
umount $MNT
The test was run a 12 cores box with 64G of ram, using a non-debug kernel config (Debian's default config) and a spinning disk.
Result before the patchset:
READ: bw=407MiB/s (427MB/s), 407MiB/s-407MiB/s (427MB/s-427MB/s), io=119GiB (128GB), run=300175-300175msec
WRITE: bw=407MiB/s (427MB/s), 407MiB/s-407MiB/s (427MB/s-427MB/s), io=119GiB (128GB), run=300175-300175msec
Result after the patchset:
READ: bw=436MiB/s (457MB/s), 436MiB/s-436MiB/s (457MB/s-457MB/s), io=128GiB (137GB), run=300044-300044msec
WRITE: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=128GiB (137GB), run=300044-300044msec
That's about +7.2% throughput for reads and +6.9% for writes.
--- fs/btrfs/ctree.h | 2 +- fs/btrfs/delalloc-space.c | 9 +++++---- fs/btrfs/file.c | 2 +- fs/btrfs/inode.c | 13 +++++++++---- fs/btrfs/qgroup.c | 5 +++-- fs/btrfs/qgroup.h | 12 ++++++++---- fs/btrfs/relocation.c | 3 ++- fs/btrfs/root-tree.c | 3 ++- 8 files changed, 31 insertions(+), 18 deletions(-)
fs/btrfs/ctree.h +1 -1
@@ -2892,7 +2892,7 @@ void btrfs_subvolume_release_metadata(struct btrfs_root *root,void btrfs_delalloc_release_extents (struct btrfs_inode *inode,u64 num_bytes);int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes, -u64 disk_num_bytes); +u64 disk_num_bytes,bool noflush);u64 btrfs_account_ro_block_groups_free_space (struct btrfs_space_info *sinfo);int btrfs_error_unpin_extent_range (struct btrfs_fs_info *fs_info,u64 start,u64 end);
fs/btrfs/delalloc-space.c +5 -4
@@ -289,7 +289,7 @@ static void calc_inode_reservations(struct btrfs_fs_info *fs_info, }int btrfs_delalloc_reserve_metadata (struct btrfs_inode *inode,u64 num_bytes, -u64 disk_num_bytes) +u64 disk_num_bytes,bool noflush) {struct btrfs_root *root = inode->root;struct btrfs_fs_info *fs_info = root->fs_info;
@@ -308,7 +308,7 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes,* If we have a transaction open (can happen if we call truncate_block * from truncate), then we need FLUSH_LIMIT so we don't deadlock. */ -if (btrfs_is_free_space_inode (inode)) { +if (noflush ||btrfs_is_free_space_inode (inode)) { flush =BTRFS_RESERVE_NO_FLUSH ; }else {if (current->journal_info)
@@ -333,7 +333,8 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes,*/ calc_inode_reservations (fs_info, num_bytes, disk_num_bytes, &meta_reserve, &qgroup_reserve); - ret =btrfs_qgroup_reserve_meta_prealloc (root, qgroup_reserve, true); + ret =btrfs_qgroup_reserve_meta_prealloc (root, qgroup_reserve, true, + noflush);if (ret)return ret; ret =btrfs_reserve_metadata_bytes (fs_info, block_rsv, meta_reserve, flush);
@@ -456,7 +457,7 @@ int btrfs_delalloc_reserve_space(struct btrfs_inode *inode, ret =btrfs_check_data_free_space (inode, reserved, start, len);if (ret <0 )return ret; - ret =btrfs_delalloc_reserve_metadata (inode, len, len); + ret =btrfs_delalloc_reserve_metadata (inode, len, len, false);if (ret <0 ) {btrfs_free_reserved_data_space (inode, *reserved, start, len);extent_changeset_free (*reserved);
fs/btrfs/file.c +1 -1
@@ -1684,7 +1684,7 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb,WARN_ON (reserve_bytes ==0 ); ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), reserve_bytes, - reserve_bytes); + reserve_bytes, false);if (ret) {if (!only_release_metadata) btrfs_free_reserved_data_space(BTRFS_I(inode),
fs/btrfs/inode.c +9 -4
@@ -4711,7 +4711,7 @@ int btrfs_truncate_block(struct btrfs_inode *inode, loff_t from, loff_t len, goto out; } } - ret =btrfs_delalloc_reserve_metadata (inode, blocksize, blocksize); + ret =btrfs_delalloc_reserve_metadata (inode, blocksize, blocksize, false);if (ret <0 ) {if (!only_release_metadata) btrfs_free_reserved_data_space(inode, data_reserved,
@@ -7420,6 +7420,7 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map, u64 start, u64 len, unsigned int iomap_flags) { + const bool nowait = (iomap_flags & IOMAP_NOWAIT);struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_map *em = *map;int type;
@@ -7457,12 +7458,15 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map,struct extent_map *em2;/* We can NOCOW, so only need to reserve metadata space. */ - ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), len, len); + ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), len, len, + nowait);if (ret <0 ) {/* Our caller expects us to free the input extent map. */ free_extent_map (em); *map =NULL ;btrfs_dec_nocow_writers (fs_info, block_start); +if (nowait && (ret == -ENOSPC || ret == -EDQUOT )) + ret = -EAGAIN ; goto out; } space_reserved = true;
@@ -7488,7 +7492,7 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map,free_extent_map (em); *map =NULL ; -if (iomap_flags &IOMAP_NOWAIT ) +if (nowait)return -EAGAIN ;/* We have to COW, so need to reserve metadata and data space. */
@@ -10813,7 +10817,8 @@ ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from, ret =btrfs_qgroup_reserve_data (inode, &data_reserved, start, num_bytes);if (ret) goto out_free_data_space; - ret =btrfs_delalloc_reserve_metadata (inode, num_bytes, disk_num_bytes); + ret =btrfs_delalloc_reserve_metadata (inode, num_bytes, disk_num_bytes, + false);if (ret) goto out_qgroup_free_data;
fs/btrfs/qgroup.c +3 -2
@@ -3939,12 +3939,13 @@ int btrfs_qgroup_reserve_meta(struct btrfs_root *root, int num_bytes, }int __btrfs_qgroup_reserve_meta (struct btrfs_root *root,int num_bytes, -enum btrfs_qgroup_rsv_type type,bool enforce) +enum btrfs_qgroup_rsv_type type,bool enforce, +bool noflush) {int ret; ret =btrfs_qgroup_reserve_meta (root, num_bytes, type, enforce); -if (ret <=0 && ret != -EDQUOT ) +if ((ret <=0 && ret != -EDQUOT ) || noflush)return ret; ret =try_flush_qgroup (root);
fs/btrfs/qgroup.h +8 -4
@@ -364,19 +364,23 @@ int btrfs_qgroup_free_data(struct btrfs_inode *inode,int btrfs_qgroup_reserve_meta (struct btrfs_root *root,int num_bytes,enum btrfs_qgroup_rsv_type type,bool enforce);int __btrfs_qgroup_reserve_meta (struct btrfs_root *root,int num_bytes, -enum btrfs_qgroup_rsv_type type,bool enforce); +enum btrfs_qgroup_rsv_type type,bool enforce, +bool noflush);/* Reserve metadata space for pertrans and prealloc type */ static inline int btrfs_qgroup_reserve_meta_pertrans (struct btrfs_root *root,int num_bytes,bool enforce) {return __btrfs_qgroup_reserve_meta (root, num_bytes, -BTRFS_QGROUP_RSV_META_PERTRANS , enforce); +BTRFS_QGROUP_RSV_META_PERTRANS , + enforce, false); }static inline int btrfs_qgroup_reserve_meta_prealloc (struct btrfs_root *root, -int num_bytes,bool enforce) +int num_bytes,bool enforce, +bool noflush) {return __btrfs_qgroup_reserve_meta (root, num_bytes, -BTRFS_QGROUP_RSV_META_PREALLOC , enforce); +BTRFS_QGROUP_RSV_META_PREALLOC , + enforce, noflush); } void __btrfs_qgroup_free_meta(struct btrfs_root *root, int num_bytes,
fs/btrfs/relocation.c +2 -1
@@ -2997,7 +2997,8 @@ static int relocate_one_page(struct inode *inode, struct file_ra_state *ra,/* Reserve metadata for this range */ ret =btrfs_delalloc_reserve_metadata (BTRFS_I (inode), - clamped_len, clamped_len); + clamped_len, clamped_len, + false);if (ret) goto release_page;
fs/btrfs/root-tree.c +2 -1
@@ -509,7 +509,8 @@ int btrfs_subvolume_reserve_metadata(struct btrfs_root *root,/* One for parent inode, two for dir entries */ qgroup_num_bytes =3 * fs_info->nodesize; ret =btrfs_qgroup_reserve_meta_prealloc (root, - qgroup_num_bytes, true); + qgroup_num_bytes, true, + false);if (ret)return ret; }
Signature · 1 line
2.33.0
Quoted 9 lines · On Wed, Mar 23, 2022 at 04:19:22PM +0000, [email protected] wrote:
From: Filipe Manana <[email protected]>
This patchset makes our direct IO code behave better for NOWAIT writes, avoiding blocking in several places for potentially long periods due to waits for IO. It also removes running the same nocow checks twice (which can be expensive) and doing extra path allocations. The last patch in the series has a test and the results I got before and after applying this patchset.
Quoted 9 lines
Filipe Manana (8):
btrfs: avoid blocking on page locks with nowait dio on compressed range
btrfs: avoid blocking nowait dio when locking file range
btrfs: avoid double nocow check when doing nowait dio writes
btrfs: stop allocating a path when checking if cross reference exists
btrfs: free path atcan_nocow_extent()before checking for checksum items
btrfs: release path earlier atcan_nocow_extent()
btrfs: avoid blocking when allocating context for nowait dio read/write
btrfs: avoid blocking on space revervation when doing nowait dio writes
Added to misc-next, thanks.