btrfs: use btrfs_path::reada to replace the
cover btrfs: use btrfs_path::reada to replace the
[PROBLEMS]
The metadata readahead code is introduced in 2011 (surprisingly, the commit message even contains changelog), but now there is only one user for it, and even for the only one user, the readahead mechanism can't provide all it needs:
- No support for commit tree readahead
Only support readahead on current tree.
- Bad layer separation
To manage on-fly bios, btrfs_reada_add() mechanism internally manages a kinda complex zone system, and it's bind to per-device.
This is against the common layer separation, as metadata should all be in btrfs logical address space, should not be bound to device physical layer.
In fact, this is the cause of all recent reada related bugs.
- No caller for asynchronous metadata readahead
Even btrfs_reada_add() is designed to be fully asynchronous, scrub only calls it in a synchronous way (call btrfs_reada_add() and immediately call btrfs_reada_wait()).
Thus rendering a lot of code unnecessary.
[ALTERNATIVE]
On the other hand, we have btrfs_path::reada mechanism, which is already used by tons of btrfs sub-systems like send.
[MODIFICATION]
This patch will use btrfs_path::reada to replace btrfs_reada_add() mechanism.
[BENCHMARK]
The conclusion looks like this:
For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.
The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.
Old New Diff
SSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%
[BENCHMARK DETAILS]
Both tests are done in the same host and VM, the VM has one dedicated SSD and one dedicated HDD attached to it (virtio driver though)
Host:
CPU: 5900X
RAM: DDR4 3200MT, no ECC
During the benchmark, there is no other active other than light DE operations.
VM:
vCPU: 16x
RAM: 4G
The VM kernel doesn't have any heavy debug options to screw up the benchmark result.
Test drives:
SSD: 500G SATA SSD
Crucial CT500MX500SSD1 (With quite some wear, as it's my main test drive for fstests)
HDD: 2T 5400rpm SATA HDD (device-managed SMR)
WDC WD20SPZX-22UA7T0 (Very new, invested just for this benchmark)
Filesystem contents:
For filesystems on SSD and HDD, they are generated using the following fio job file:
[scrub-populate]
directory=/mnt/btrfs
nrfiles=16384
openfiles=16
filesize=2k-512k
readwrite=randwrite
ioengine=libaio
fallocate=none
numjobs=4
Then randomly remove 4096 files (1/16th of total files) to create enough gaps in extent tree.
Finally run scrub on each filesystem 5 times, with cycle mount and module reload between each run.
Full result can be fetched here:
https://docs.google.com/spreadsheets/d/1cwUAlbKPfp4baKrS92debsCt6Ejqvxr_Ylspj_SDFT0/edit?usp=sharing
[CHANGELOG]RFC->v1:
- Add benchmark result
- Add extent tree readahead using btrfs_path::reada
Qu Wenruo (3):
btrfs: remove the unnecessary path parameter for scrub_raid56_parity()
btrfs: remove reada mechanism
btrfs: use btrfs_path::reada for scrub extent tree readahead
fs/btrfs/Makefile | 2 +- fs/btrfs/ctree.h | 25 - fs/btrfs/dev-replace.c | 5 - fs/btrfs/disk-io.c | 20 +- fs/btrfs/extent_io.c | 3 - fs/btrfs/reada.c | 1086 ---------------------------------------- fs/btrfs/scrub.c | 65 +-- fs/btrfs/super.c | 1 - fs/btrfs/volumes.c | 7 - fs/btrfs/volumes.h | 7 - 10 files changed, 18 insertions(+), 1203 deletions(-) delete mode 100644 fs/btrfs/reada.c
Signature · 1 line
2.34.1
PATCH 1/3 btrfs: remove the unnecessary path parameter for scrub_raid56_parity()
In function scrub_stripe() we allocated two btrfs_path, one @path for extent tree search and another @ppath for full stripe extent tree search for RAID56.
This is totally uncessary, as the @ppath usage is completely inside scrub_raid56_parity(), thus we can move the path allocation into scrub_raid56_parity() completely.
--- fs/btrfs/scrub.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
fs/btrfs/scrub.c +14 -14
@@ -2892,7 +2892,6 @@ static void scrub_parity_put(struct scrub_parity *sparity) static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,struct map_lookup *map,struct btrfs_device *sdev, -struct btrfs_path *path,u64 logic_start,u64 logic_end) {
@@ -2901,6 +2900,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,struct btrfs_root *csum_root;struct btrfs_extent_item *extent;struct btrfs_io_context *bioc =NULL ; +struct btrfs_path *path;u64 flags;int ret;int slot;
@@ -2919,6 +2919,14 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,int extent_mirror_num;int stop_loop =0 ; + path =btrfs_alloc_path (); +if (!path) { +spin_lock (&sctx->stat_lock); + sctx->stat.malloc_errors++; +spin_unlock (&sctx->stat_lock); +return -ENOMEM ; + } +ASSERT (map->stripe_len <=U32_MAX ); nsectors = map->stripe_len >> fs_info->sectorsize_bits; bitmap_len =scrub_calc_parity_bitmap_len (nsectors);
@@ -2928,6 +2936,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,spin_lock (&sctx->stat_lock); sctx->stat.malloc_errors++;spin_unlock (&sctx->stat_lock); +btrfs_free_path (path);return -ENOMEM ; }
@@ -3117,7 +3126,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,scrub_wr_submit (sctx);mutex_unlock (&sctx->wr_lock); -btrfs_release_path (path); +btrfs_free_path (path);return ret <0 ? ret :0 ; }
@@ -3167,7 +3176,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, int num, u64 base, u64 length, struct btrfs_block_group *cache) { - struct btrfs_path *path, *ppath; + struct btrfs_path *path;struct btrfs_fs_info *fs_info = sctx->fs_info;struct btrfs_root *root;struct btrfs_root *csum_root;
@@ -3229,12 +3238,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,if (!path)return -ENOMEM ; - ppath =btrfs_alloc_path (); -if (!ppath) { -btrfs_free_path (path); -return -ENOMEM ; - } -/* * work on commit root. The related disk blocks are static as * long as COW is applied. This means, it is save to rewrite
@@ -3243,8 +3246,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, path->search_commit_root =1 ; path->skip_locking =1 ; - ppath->search_commit_root =1 ; - ppath->skip_locking =1 ;/* * trigger the readahead for extent tree csum tree and wait for * completion. During readahead, the scrub is officially paused
@@ -3347,7 +3348,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, stripe_logical += base; stripe_end = stripe_logical + increment; ret =scrub_raid56_parity (sctx, map, scrub_dev, - ppath, stripe_logical, + stripe_logical, stripe_end);if (ret) goto out;
@@ -3518,7 +3519,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, stripe_end = stripe_logical + increment; ret =scrub_raid56_parity (sctx, - map, scrub_dev, ppath, + map, scrub_dev, stripe_logical, stripe_end);if (ret)
@@ -3565,7 +3566,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,blk_finish_plug (&plug);btrfs_free_path (path); -btrfs_free_path (ppath);if (sctx->is_dev_replace && ret >=0 ) {int ret2;
Signature · 1 line
2.34.1
Re: [PATCH 1/3] btrfs: remove the unnecessary path parameter for scrub_raid56_parity()
Quoted 88 lines · Qu Wenruo
In function
scrub_stripe()we allocated twobtrfs_path, one @path for extent tree search and another @ppath for full stripe extent tree search for RAID56.This is totally uncessary, as the @ppath usage is completely inside
scrub_raid56_parity(), thus we can move the path allocation intoscrub_raid56_parity()completely.Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/scrub.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)fs/btrfs/scrub.c +12
-9@@ -2892,7 +2892,6 @@ static void scrub_parity_put(struct scrub_parity *sparity) static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,struct map_lookup *map,struct btrfs_device *sdev, -struct btrfs_path *path,u64 logic_start,u64 logic_end) {@@ -2901,6 +2900,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,struct btrfs_root *csum_root;struct btrfs_extent_item *extent;struct btrfs_io_context *bioc =NULL ; +struct btrfs_path *path;u64 flags;int ret;int slot;@@ -2919,6 +2919,14 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,int extent_mirror_num;int stop_loop =0 ; + path =btrfs_alloc_path (); +if (!path) { +spin_lock (&sctx->stat_lock); + sctx->stat.malloc_errors++; +spin_unlock (&sctx->stat_lock); +return -ENOMEM ; + } +ASSERT (map->stripe_len <=U32_MAX ); nsectors = map->stripe_len >> fs_info->sectorsize_bits; bitmap_len =scrub_calc_parity_bitmap_len (nsectors);@@ -2928,6 +2936,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,spin_lock (&sctx->stat_lock); sctx->stat.malloc_errors++;spin_unlock (&sctx->stat_lock); +btrfs_free_path (path);return -ENOMEM ; }@@ -3117,7 +3126,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,scrub_wr_submit (sctx);mutex_unlock (&sctx->wr_lock); -btrfs_release_path (path); +btrfs_free_path (path);return ret <0 ? ret :0 ; }@@ -3167,7 +3176,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, int num, u64 base, u64 length, struct btrfs_block_group *cache) { - struct btrfs_path *path, *ppath; + struct btrfs_path *path;struct btrfs_fs_info *fs_info = sctx->fs_info;struct btrfs_root *root;struct btrfs_root *csum_root;@@ -3229,12 +3238,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,if (!path)return -ENOMEM ; - ppath =btrfs_alloc_path (); -if (!ppath) { -btrfs_free_path (path); -return -ENOMEM ; - } -/* * work on commit root. The related disk blocks are static as * long as COW is applied. This means, it is save to rewrite
fs/btrfs/scrub.c static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, · line 3247
Quoted 5 lines
fs/btrfs/scrub.c +0
-2@@ -3243,8 +3246,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, path->search_commit_root =1 ; path->skip_locking =1 ; - ppath->search_commit_root =1 ; - ppath->skip_locking =1 ;
The new path in the raid56 functions doesn't have this.
This is causing false alerts in raid56 dev-replace test cases, like btrfs/072. Which reports csum mismatch for metadata.
Would fix it in next update.
Thanks,
Qu
Quoted 29 lines
fs/btrfs/scrub.c +2
-3/* * trigger the readahead for extent tree csum tree and wait for * completion. During readahead, the scrub is officially paused @@ -3347,7 +3348,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, stripe_logical += base; stripe_end = stripe_logical + increment; ret =scrub_raid56_parity (sctx, map, scrub_dev, - ppath, stripe_logical, + stripe_logical, stripe_end);if (ret) goto out;@@ -3518,7 +3519,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, stripe_end = stripe_logical + increment; ret =scrub_raid56_parity (sctx, - map, scrub_dev, ppath, + map, scrub_dev, stripe_logical, stripe_end);if (ret)@@ -3565,7 +3566,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,blk_finish_plug (&plug);btrfs_free_path (path); -btrfs_free_path (ppath);if (sctx->is_dev_replace && ret >=0 ) {int ret2;
PATCH 2/3 btrfs: remove reada mechanism
In reply to Qu WenruoCurrently there is only one user for btrfs metadata readahead, and that's scrub.
But even for the single user, it's not providing the correct functionality it needs, as scrub needs reada for commit root, which current readahead can't provide. (Although it's pretty easy to add such feature).
Despite this, there are some extra problems related to metadata readahead:
- Duplicated feature with btrfs_path::reada
- Partly duplicated feature of btrfs_fs_info::buffer_radix Btrfs already caches its metadata in buffer_radix, while readahead tries to read the tree block no matter if it's already cached.
- Poor layer separation
To my surprise, metadata readahead works kinda at device level. This is definitely not the correct layer it should be, since metadata is at btrfs logical address space, it should not bother device at all.
This brings extra chance for bugs to sneak in, while brings unnecessary complexity.
- Dead codes
In the very beginning of scrub.c we have #undef DEBUG, rendering all the debug related code useless and unable to test.
Thus here I purpose to remove the metadata readahead mechanism completely.
--- fs/btrfs/Makefile | 2 +- fs/btrfs/ctree.h | 25 - fs/btrfs/dev-replace.c | 5 - fs/btrfs/disk-io.c | 20 +- fs/btrfs/extent_io.c | 3 - fs/btrfs/reada.c | 1086 ---------------------------------------- fs/btrfs/scrub.c | 36 -- fs/btrfs/super.c | 1 - fs/btrfs/volumes.c | 7 - fs/btrfs/volumes.h | 7 - 10 files changed, 3 insertions(+), 1189 deletions(-) delete mode 100644 fs/btrfs/reada.c
fs/btrfs/Makefile +1 -1
@@ -27,7 +27,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \ extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \ export.o tree-log.o free-space-cache.o zlib.o lzo.o zstd.o \ compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \ - reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \ + backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \ uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \ block-rsv.o delalloc-space.o block-group.o discard.o reflink.o \ subpage.o tree-mod-log.o
fs/btrfs/ctree.h +0 -25
@@ -821,7 +821,6 @@ struct btrfs_fs_info {struct btrfs_workqueue *endio_write_workers;struct btrfs_workqueue *endio_freespace_worker;struct btrfs_workqueue *caching_workers; -struct btrfs_workqueue *readahead_workers;/* * fixup workers take dirty pages that didn't properly go through
@@ -958,13 +957,6 @@ struct btrfs_fs_info {struct btrfs_delayed_root *delayed_root; -/* readahead tree */ -spinlock_t reada_lock; -struct radix_tree_root reada_tree; - -/* readahead works cnt */ -atomic_t reada_works_cnt; -/* Extent buffer radix tree */ spinlock_t buffer_lock;/* Entries are eb->start / sectorsize */
@@ -3836,23 +3828,6 @@ static inline void btrfs_bio_counter_dec(struct btrfs_fs_info *fs_info)btrfs_bio_counter_sub (fs_info,1 ); } -/* reada.c */ -struct reada_control { -struct btrfs_fs_info *fs_info;/* tree to prefetch */ -struct btrfs_key key_start; -struct btrfs_key key_end;/* exclusive */ -atomic_t elems; -struct kref refcnt; -wait_queue_head_t wait; -}; -struct reada_control *btrfs_reada_add (struct btrfs_root *root, -struct btrfs_key *start,struct btrfs_key *end); -int btrfs_reada_wait (void *handle); -void btrfs_reada_detach (void *handle); -int btree_readahead_hook (struct extent_buffer *eb,int err); -void btrfs_reada_remove_dev (struct btrfs_device *dev); -void btrfs_reada_undo_remove_dev (struct btrfs_device *dev); -static inline int is_fstree (u64 rootid) { if (rootid == BTRFS_FS_TREE_OBJECTID ||
fs/btrfs/dev-replace.c +0 -5
@@ -906,9 +906,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info, }btrfs_wait_ordered_roots (fs_info,U64_MAX ,0 , (u64)-1 ); -if (!scrub_ret) -btrfs_reada_remove_dev (src_device); -/* * We have to use this loop approach because at this point src_device * has to be available for transaction commit to complete, yet new
@@ -917,7 +914,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,while (1 ) { trans =btrfs_start_transaction (root,0 );if (IS_ERR (trans)) { -btrfs_reada_undo_remove_dev (src_device);mutex_unlock (&dev_replace->lock_finishing_cancel_unmount);return PTR_ERR (trans); }
@@ -968,7 +964,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,up_write (&dev_replace->rwsem);mutex_unlock (&fs_info->chunk_mutex);mutex_unlock (&fs_info->fs_devices->device_list_mutex); -btrfs_reada_undo_remove_dev (src_device);btrfs_rm_dev_replace_blocked (fs_info);if (tgt_device)btrfs_destroy_dev_replace_tgtdev (tgt_device);
fs/btrfs/disk-io.c +2 -18
@@ -665,9 +665,6 @@ static int validate_subpage_buffer(struct page *page, u64 start, u64 end,if (ret <0 ) goto err; -if (test_and_clear_bit (EXTENT_BUFFER_READAHEAD , &eb->bflags)) -btree_readahead_hook (eb, ret); -set_extent_buffer_uptodate (eb);free_extent_buffer (eb);
@@ -715,10 +712,6 @@ int btrfs_validate_metadata_buffer(struct btrfs_bio *bbio, } ret =validate_extent_buffer (eb); err: -if (reads_done && -test_and_clear_bit (EXTENT_BUFFER_READAHEAD , &eb->bflags)) -btree_readahead_hook (eb, ret); -if (ret) {/* * our io error hook is going to dec the io pages
@@ -2224,7 +2217,6 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)btrfs_destroy_workqueue (fs_info->endio_freespace_worker);btrfs_destroy_workqueue (fs_info->delayed_workers);btrfs_destroy_workqueue (fs_info->caching_workers); -btrfs_destroy_workqueue (fs_info->readahead_workers);btrfs_destroy_workqueue (fs_info->flush_workers);btrfs_destroy_workqueue (fs_info->qgroup_rescan_workers);if (fs_info->discard_ctl.discard_workers)
@@ -2437,9 +2429,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) fs_info->delayed_workers =btrfs_alloc_workqueue (fs_info,"delayed-meta" , flags, max_active,0 ); - fs_info->readahead_workers = -btrfs_alloc_workqueue (fs_info,"readahead" , flags, - max_active,2 ); fs_info->qgroup_rescan_workers =btrfs_alloc_workqueue (fs_info,"qgroup-rescan" , flags,1 ,0 ); fs_info->discard_ctl.discard_workers =
@@ -2451,9 +2440,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) fs_info->endio_meta_write_workers && fs_info->endio_write_workers && fs_info->endio_raid56_workers && fs_info->endio_freespace_worker && fs_info->rmw_workers && - fs_info->caching_workers && fs_info->readahead_workers && - fs_info->fixup_workers && fs_info->delayed_workers && - fs_info->qgroup_rescan_workers && + fs_info->caching_workers && fs_info->fixup_workers && + fs_info->delayed_workers && fs_info->qgroup_rescan_workers && fs_info->discard_ctl.discard_workers)) { return -ENOMEM; }
@@ -3083,7 +3071,6 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)atomic_set (&fs_info->async_delalloc_pages,0 );atomic_set (&fs_info->defrag_running,0 ); -atomic_set (&fs_info->reada_works_cnt,0 );atomic_set (&fs_info->nr_delayed_iputs,0 );atomic64_set (&fs_info->tree_mod_seq,0 ); fs_info->global_root_tree =RB_ROOT ;
@@ -3094,9 +3081,6 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info) fs_info->tree_mod_log =RB_ROOT ; fs_info->commit_interval =BTRFS_DEFAULT_COMMIT_INTERVAL ; fs_info->avg_delayed_ref_runtime =NSEC_PER_SEC >>6 ;/* div by 64 */ -/* readahead state */ -INIT_RADIX_TREE (&fs_info->reada_tree,GFP_NOFS & ~__GFP_DIRECT_RECLAIM); -spin_lock_init (&fs_info->reada_lock);btrfs_init_ref_verify (fs_info); fs_info->thread_pool_size = min_t(unsigned long,
fs/btrfs/extent_io.c +0 -3
@@ -3087,9 +3087,6 @@ static void end_bio_extent_readpage(struct bio *bio)set_bit (EXTENT_BUFFER_READ_ERR , &eb->bflags); eb->read_mirror = mirror;atomic_dec (&eb->io_pages); -if (test_and_clear_bit (EXTENT_BUFFER_READAHEAD , - &eb->bflags)) -btree_readahead_hook (eb, -EIO ); } readpage_ok:if (likely (uptodate)) {
fs/btrfs/reada.c deleted +0 -1086
1,086 lines not on this page · raw message
fs/btrfs/scrub.c +0 -36
@@ -3193,10 +3193,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,u64 physical_end;u64 generation;int mirror_num; -struct reada_control *reada1; -struct reada_control *reada2;struct btrfs_key key; -struct btrfs_key key_end;u64 increment = map->stripe_len;u64 offset;u64 extent_logical;
@@ -3246,11 +3243,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, path->search_commit_root =1 ; path->skip_locking =1 ; -/* -* trigger the readahead for extent tree csum tree and wait for -* completion. During readahead, the scrub is officially paused -* to not hold off transaction commits -*/ logical = base + offset; physical_end = physical + nstripes * map->stripe_len;if (map->type &BTRFS_BLOCK_GROUP_RAID56_MASK ) {
@@ -3265,36 +3257,8 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,scrub_blocked_if_needed (fs_info); root =btrfs_extent_root (fs_info, logical); - -/* FIXME it might be better to start readahead at commit root */ - key.objectid = logical; - key.type =BTRFS_EXTENT_ITEM_KEY ; - key.offset = (u64 )0 ; - key_end.objectid = logic_end; - key_end.type =BTRFS_METADATA_ITEM_KEY ; - key_end.offset = (u64)-1 ; - reada1 =btrfs_reada_add (root, &key, &key_end); - csum_root =btrfs_csum_root (fs_info, logical); -if (cache->flags &BTRFS_BLOCK_GROUP_DATA ) { - key.objectid =BTRFS_EXTENT_CSUM_OBJECTID ; - key.type =BTRFS_EXTENT_CSUM_KEY ; - key.offset = logical; - key_end.objectid =BTRFS_EXTENT_CSUM_OBJECTID ; - key_end.type =BTRFS_EXTENT_CSUM_KEY ; - key_end.offset = logic_end; - reada2 =btrfs_reada_add (csum_root, &key, &key_end); - }else { - reada2 =NULL ; - } - -if (!IS_ERR (reada1)) -btrfs_reada_wait (reada1); -if (!IS_ERR_OR_NULL (reada2)) -btrfs_reada_wait (reada2); - -/* * collect all data csums for the stripe to avoid seeking during * the scrub. This might currently (crc32) end up to be about 1MB
fs/btrfs/super.c +0 -1
@@ -1842,7 +1842,6 @@ static void btrfs_resize_thread_pool(struct btrfs_fs_info *fs_info,btrfs_workqueue_set_max (fs_info->endio_write_workers, new_pool_size);btrfs_workqueue_set_max (fs_info->endio_freespace_worker, new_pool_size);btrfs_workqueue_set_max (fs_info->delayed_workers, new_pool_size); -btrfs_workqueue_set_max (fs_info->readahead_workers, new_pool_size);btrfs_workqueue_set_max (fs_info->scrub_wr_completion_workers, new_pool_size); }
fs/btrfs/volumes.c +0 -7
@@ -1166,7 +1166,6 @@ static void btrfs_close_one_device(struct btrfs_device *device)ASSERT (!test_bit (BTRFS_DEV_STATE_REPLACE_TGT , &device->dev_state));ASSERT (list_empty (&device->dev_alloc_list));ASSERT (list_empty (&device->post_commit_list)); -ASSERT (atomic_read (&device->reada_in_flight) ==0 ); }static void close_fs_devices (struct btrfs_fs_devices *fs_devices)
@@ -2148,8 +2147,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, } ret =btrfs_shrink_device (device,0 ); -if (!ret) -btrfs_reada_remove_dev (device);if (ret) goto error_undo;
@@ -2247,7 +2244,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info,return ret; error_undo: -btrfs_reada_undo_remove_dev (device);if (test_bit (BTRFS_DEV_STATE_WRITEABLE , &device->dev_state)) {mutex_lock (&fs_info->chunk_mutex); list_add(&device->dev_alloc_list,
@@ -6978,11 +6974,8 @@ struct btrfs_device *btrfs_alloc_device(struct btrfs_fs_info *fs_info,INIT_LIST_HEAD (&dev->dev_alloc_list);INIT_LIST_HEAD (&dev->post_commit_list); -atomic_set (&dev->reada_in_flight,0 );atomic_set (&dev->dev_stats_ccnt,0 );btrfs_device_data_ordered_init (dev); -INIT_RADIX_TREE (&dev->reada_zones,GFP_NOFS & ~__GFP_DIRECT_RECLAIM); -INIT_RADIX_TREE (&dev->reada_extents,GFP_NOFS & ~__GFP_DIRECT_RECLAIM);extent_io_tree_init (fs_info, &dev->alloc_state,IO_TREE_DEVICE_ALLOC_STATE ,NULL );
fs/btrfs/volumes.h +0 -7
@@ -123,13 +123,6 @@ struct btrfs_device {/* per-device scrub information */ struct scrub_ctx *scrub_ctx; -/* readahead state */ -atomic_t reada_in_flight; -u64 reada_next; -struct reada_zone *reada_curr_zone; -struct radix_tree_root reada_zones; -struct radix_tree_root reada_extents; -/* disk I/O failure stats. For detailed description refer to * enum btrfs_dev_stat_values in ioctl.h */ int dev_stats_valid;
Signature · 1 line
2.34.1
PATCH 3/3 btrfs: use btrfs_path::reada for scrub extent tree readahead
In reply to Qu WenruoFor scrub, we trigger two readahead for two trees, extent tree to get where to scrub, and csum tree to get the data checksum.
For csum tree we already trigger readahead in btrfs_lookup_csums_range(), by setting path->reada. But for extent tree we don't have any path based readahead.
This patch will add the readahead for extent tree.
With path based readahead only, on SSDs the scrub speed is the same as previous btrfs_reada_add() mechanism.
While on HDDs, the scrub speed get degraded by at most 5%, although not the best case, it's still pretty acceptable, especially considering how many bugs there are in the old btrfs_reada_add() mechanism recently.
--- fs/btrfs/scrub.c | 1 + 1 file changed, 1 insertion(+)
fs/btrfs/scrub.c +1 -0
@@ -3242,6 +3242,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,*/ path->search_commit_root =1 ; path->skip_locking =1 ; + path->reada =READA_FORWARD ; logical = base + offset; physical_end = physical + nstripes * map->stripe_len;
Signature · 1 line
2.34.1
Quoted 17 lines · Qu Wenruo
[PROBLEMS]
The metadata readahead code is introduced in 2011 (surprisingly, the commit message even contains changelog), but now there is only one user for it, and even for the only one user, the readahead mechanism can't provide all it needs:- No support for commit tree readahead
Only support readahead on current tree.- Bad layer separation
To manage on-fly bios,btrfs_reada_add()mechanism internally manages a kinda complex zone system, and it's bind to per-device.This is against the common layer separation, as metadata should all be in btrfs logical address space, should not be bound to device physical layer.
Quoted 7 lines
In fact, this is the cause of all recent reada related bugs.
- No caller for asynchronous metadata readahead
Evenbtrfs_reada_add()is designed to be fully asynchronous, scrub only calls it in a synchronous way (callbtrfs_reada_add()and immediately callbtrfs_reada_wait()).
Thus rendering a lot of code unnecessary.
I agree with removing the reada.c code, it's overengineered perhaps with intentions to use it in more places but this hasn't happened and nobody is interested doing the work. We have the path readahead so it's not we'd lose readahead capabilities at all.
Thanks for benchmarking it, the drop is acceptable and we know people are more interested in limiting the load anyway.
Quoted 8 lines
[BENCHMARK]
The conclusion looks like this:For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.
Quoted 7 lines
The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.Old New DiffSSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%
I'll copy this information to the last patch changelog.
Quoted 46 lines · David Sterba
Quoted 17 lines · Qu Wenruo
[PROBLEMS]
The metadata readahead code is introduced in 2011 (surprisingly, the commit message even contains changelog), but now there is only one user for it, and even for the only one user, the readahead mechanism can't provide all it needs:- No support for commit tree readahead
Only support readahead on current tree.- Bad layer separation
To manage on-fly bios,btrfs_reada_add()mechanism internally manages a kinda complex zone system, and it's bind to per-device.This is against the common layer separation, as metadata should all be in btrfs logical address space, should not be bound to device physical layer.
Quoted 7 lines
In fact, this is the cause of all recent reada related bugs.
- No caller for asynchronous metadata readahead
Evenbtrfs_reada_add()is designed to be fully asynchronous, scrub only calls it in a synchronous way (callbtrfs_reada_add()and immediately callbtrfs_reada_wait()).
Thus rendering a lot of code unnecessary.I agree with removing the
reada.ccode, it's overengineered perhaps with intentions to use it in more places but this hasn't happened and nobody is interested doing the work. We have the path readahead so it's not we'd lose readahead capabilities at all.Thanks for benchmarking it, the drop is acceptable and we know people are more interested in limiting the load anyway.
Quoted 12 lines
[BENCHMARK]
The conclusion looks like this:For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.
Quoted 5 lines
Quoted 3 lines
Old New DiffSSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%I'll copy this information to the last patch changelog.
Since I found a bug in the first patch, let me have the chance to re-order the patches, and put this into the patch removing reada.
Thanks,
Qu