Skip to contentSkip to conversation

btrfs: use btrfs_path::reada to replace the under-utilized btrfs_reada_add() mechanism

PATCH In review Qu Wenruo 2 participants 5 messages Dec 14, 2021 – Dec 15, 2021 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v2 cover btrfs: use btrfs_path::reada to replace the under-utilized btrfs_reada_add() mechanism

[PROBLEMS]
The metadata readahead code is introduced in 2011 (surprisingly, the commit message even contains changelog), but now there is only one user for it, and even for the only one user, the readahead mechanism can't provide all it needs:

- No support for commit tree readahead
Only support readahead on current tree.

- Bad layer separation
To manage on-fly bios, btrfs_reada_add() mechanism internally manages a kinda complex zone system, and it's bind to per-device.

This is against the common layer separation, as metadata should all be in btrfs logical address space, should not be bound to device physical layer.

In fact, this is the cause of all recent reada related bugs.

- No caller for asynchronous metadata readahead
Even btrfs_reada_add() is designed to be fully asynchronous, scrub only calls it in a synchronous way (call btrfs_reada_add() and immediately call btrfs_reada_wait()).
Thus rendering a lot of code unnecessary.

[ALTERNATIVE]
On the other hand, we have btrfs_path::reada mechanism, which is already used by tons of btrfs sub-systems like send.

[MODIFICATION]
This patch will use btrfs_path::reada to replace btrfs_reada_add() mechanism.

[BENCHMARK]
The conclusion looks like this:

For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.

The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.

	Old		New		Diff

SSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%

[BENCHMARK DETAILS]
Both tests are done in the same host and VM, the VM has one dedicated SSD and one dedicated HDD attached to it (virtio driver though)

Host:
CPU: 5900X
RAM: DDR4 3200MT, no ECC

During the benchmark, there is no other active other than light DE operations.

VM:
vCPU: 16x
RAM: 4G

The VM kernel doesn't have any heavy debug options to screw up the benchmark result.

Test drives:
SSD: 500G SATA SSD

	Crucial CT500MX500SSD1
	(With quite some wear, as it's my main test drive for fstests)

HDD: 2T 5400rpm SATA HDD (device-managed SMR)

	WDC WD20SPZX-22UA7T0
	(Very new, invested just for this benchmark)

Filesystem contents:
For filesystems on SSD and HDD, they are generated using the following fio job file:

[scrub-populate]
directory=/mnt/btrfs
nrfiles=16384
openfiles=16
filesize=2k-512k
readwrite=randwrite
ioengine=libaio
fallocate=none
numjobs=4

Then randomly remove 4096 files (1/16th of total files) to create enough gaps in extent tree.

Finally run scrub on each filesystem 5 times, with cycle mount and module reload between each run.

Full result can be fetched here:
https://docs.google.com/spreadsheets/d/1cwUAlbKPfp4baKrS92debsCt6Ejqvxr_Ylspj_SDFT0/edit?usp=sharing

[CHANGELOG]
RFC->v1:
- Add benchmark result
- Add extent tree readahead using btrfs_path::reada

v2:
- Reorder the patches
So that the reada removal comes at last

- Add benchmark result into the reada removal patch

- Fix a bug that can cause false alert for RAID56 dev-replace/scrub Caused by missing ->search_commit_root assignment during refactor.

Qu Wenruo (3):
btrfs: remove the unnecessary path parameter for scrub_raid56_parity()
btrfs: use btrfs_path::reada for scrub extent tree readahead
btrfs: remove reada mechanism

 fs/btrfs/Makefile      |    2 +-
 fs/btrfs/ctree.h       |   25 -
 fs/btrfs/dev-replace.c |    5 -
 fs/btrfs/disk-io.c     |   20 +-
 fs/btrfs/extent_io.c   |    3 -
 fs/btrfs/reada.c       | 1086 ----------------------------------------
 fs/btrfs/scrub.c       |   67 +--
 fs/btrfs/super.c       |    1 -
 fs/btrfs/volumes.c     |    7 -
 fs/btrfs/volumes.h     |    7 -
 10 files changed, 20 insertions(+), 1203 deletions(-)
 delete mode 100644 fs/btrfs/reada.c
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 1/3 btrfs: remove the unnecessary path parameter for scrub_raid56_parity()

In function scrub_stripe() we allocated two btrfs_path, one @path for extent tree search and another @ppath for full stripe extent tree search for RAID56.

This is totally uncessary, as the @ppath usage is completely inside scrub_raid56_parity(), thus we can move the path allocation into scrub_raid56_parity() completely.

---
 fs/btrfs/scrub.c | 30 ++++++++++++++++--------------
 1 file changed, 16 insertions(+), 14 deletions(-)
fs/btrfs/scrub.c +16 -14
index 15a123e67108..41908f0c1e76 100644
@@ -2892,7 +2892,6 @@ static void scrub_parity_put(struct scrub_parity *sparity)
 static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 						  struct map_lookup *map,
 						  struct btrfs_device *sdev,
-						  struct btrfs_path *path,
 						  u64 logic_start,
 						  u64 logic_end)
 {
@@ -2901,6 +2900,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 	struct btrfs_root *csum_root;
 	struct btrfs_extent_item *extent;
 	struct btrfs_io_context *bioc = NULL;
+	struct btrfs_path *path;
 	u64 flags;
 	int ret;
 	int slot;
@@ -2919,6 +2919,16 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 	int extent_mirror_num;
 	int stop_loop = 0;
 
+	path = btrfs_alloc_path();
+	if (!path) {
+		spin_lock(&sctx->stat_lock);
+		sctx->stat.malloc_errors++;
+		spin_unlock(&sctx->stat_lock);
+		return -ENOMEM;
+	}
+	path->search_commit_root = 1;
+	path->skip_locking = 1;
+
 	ASSERT(map->stripe_len <= U32_MAX);
 	nsectors = map->stripe_len >> fs_info->sectorsize_bits;
 	bitmap_len = scrub_calc_parity_bitmap_len(nsectors);
@@ -2928,6 +2938,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 		spin_lock(&sctx->stat_lock);
 		sctx->stat.malloc_errors++;
 		spin_unlock(&sctx->stat_lock);
+		btrfs_free_path(path);
 		return -ENOMEM;
 	}
 
@@ -3117,7 +3128,7 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx,
 	scrub_wr_submit(sctx);
 	mutex_unlock(&sctx->wr_lock);
 
-	btrfs_release_path(path);
+	btrfs_free_path(path);
 	return ret < 0 ? ret : 0;
 }
 
@@ -3167,7 +3178,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 					   int num, u64 base, u64 length,
 					   struct btrfs_block_group *cache)
 {
-	struct btrfs_path *path, *ppath;
+	struct btrfs_path *path;
 	struct btrfs_fs_info *fs_info = sctx->fs_info;
 	struct btrfs_root *root;
 	struct btrfs_root *csum_root;
@@ -3229,12 +3240,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	if (!path)
 		return -ENOMEM;
 
-	ppath = btrfs_alloc_path();
-	if (!ppath) {
-		btrfs_free_path(path);
-		return -ENOMEM;
-	}
-
 	/*
 	 * work on commit root. The related disk blocks are static as
 	 * long as COW is applied. This means, it is save to rewrite
@@ -3243,8 +3248,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	path->search_commit_root = 1;
 	path->skip_locking = 1;
 
-	ppath->search_commit_root = 1;
-	ppath->skip_locking = 1;
 	/*
 	 * trigger the readahead for extent tree csum tree and wait for
 	 * completion. During readahead, the scrub is officially paused
@@ -3347,7 +3350,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 				stripe_logical += base;
 				stripe_end = stripe_logical + increment;
 				ret = scrub_raid56_parity(sctx, map, scrub_dev,
-							  ppath, stripe_logical,
+							  stripe_logical,
 							  stripe_end);
 				if (ret)
 					goto out;
@@ -3518,7 +3521,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 						stripe_end = stripe_logical +
 								increment;
 						ret = scrub_raid56_parity(sctx,
-							map, scrub_dev, ppath,
+							map, scrub_dev,
 							stripe_logical,
 							stripe_end);
 						if (ret)
@@ -3565,7 +3568,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 
 	blk_finish_plug(&plug);
 	btrfs_free_path(path);
-	btrfs_free_path(ppath);
 
 	if (sctx->is_dev_replace && ret >= 0) {
 		int ret2;
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v2 2/3 btrfs: use btrfs_path::reada for scrub extent tree readahead

In reply to Qu Wenruo

For scrub, we trigger two readahead for two trees, extent tree to get where to scrub, and csum tree to get the data checksum.

For csum tree we already trigger readahead in btrfs_lookup_csums_range(), by setting path->reada. But for extent tree we don't have any path based readahead.

This patch will add the readahead for extent tree, so we can later remove the btrfs_reada_add() based readahead.

---
 fs/btrfs/scrub.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/scrub.c +1 -0
index 41908f0c1e76..70457aadd721 100644
@@ -3247,6 +3247,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	 */
 	path->search_commit_root = 1;
 	path->skip_locking = 1;
+	path->reada = READA_FORWARD;
 
 	/*
 	 * trigger the readahead for extent tree csum tree and wait for
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v2 3/3 btrfs: remove reada mechanism

In reply to Qu Wenruo

Currently there is only one user for btrfs metadata readahead, and that's scrub.

But even for the single user, it's not providing the correct functionality it needs, as scrub needs reada for commit root, which current readahead can't provide. (Although it's pretty easy to add such feature).

Despite this, there are some extra problems related to metadata readahead:

- Duplicated feature with btrfs_path::reada

- Partly duplicated feature of btrfs_fs_info::buffer_radix Btrfs already caches its metadata in buffer_radix, while readahead tries to read the tree block no matter if it's already cached.

- Poor layer separation
To my surprise, metadata readahead works kinda at device level. This is definitely not the correct layer it should be, since metadata is at btrfs logical address space, it should not bother device at all.

This brings extra chance for bugs to sneak in, while brings unnecessary complexity.

- Dead codes
In the very beginning of scrub.c we have #undef DEBUG, rendering all the debug related code useless and unable to test.

Thus here I purpose to remove the metadata readahead mechanism completely.

[BENCHMARK]
There is a full benchmark for the scrub performance difference using the old btrfs_reada_add() and btrfs_path::reada.

For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.

The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.

	Old		New		Diff

SSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%

Comprehensive test methodology is in the cover letter of the patch.

---
 fs/btrfs/Makefile      |    2 +-
 fs/btrfs/ctree.h       |   25 -
 fs/btrfs/dev-replace.c |    5 -
 fs/btrfs/disk-io.c     |   20 +-
 fs/btrfs/extent_io.c   |    3 -
 fs/btrfs/reada.c       | 1086 ----------------------------------------
 fs/btrfs/scrub.c       |   36 --
 fs/btrfs/super.c       |    1 -
 fs/btrfs/volumes.c     |    7 -
 fs/btrfs/volumes.h     |    7 -
 10 files changed, 3 insertions(+), 1189 deletions(-)
 delete mode 100644 fs/btrfs/reada.c
fs/btrfs/Makefile +1 -1
index 3dcf9bcc2326..4188ba3fd8c3 100644
@@ -27,7 +27,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
 	   export.o tree-log.o free-space-cache.o zlib.o lzo.o zstd.o \
 	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
-	   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
+	   backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
 	   uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
 	   block-rsv.o delalloc-space.o block-group.o discard.o reflink.o \
 	   subpage.o tree-mod-log.o
fs/btrfs/ctree.h +0 -25
index dfee4b403da1..eeda7b687fca 100644
@@ -821,7 +821,6 @@ struct btrfs_fs_info {
 	struct btrfs_workqueue *endio_write_workers;
 	struct btrfs_workqueue *endio_freespace_worker;
 	struct btrfs_workqueue *caching_workers;
-	struct btrfs_workqueue *readahead_workers;
 
 	/*
 	 * fixup workers take dirty pages that didn't properly go through
@@ -958,13 +957,6 @@ struct btrfs_fs_info {
 
 	struct btrfs_delayed_root *delayed_root;
 
-	/* readahead tree */
-	spinlock_t reada_lock;
-	struct radix_tree_root reada_tree;
-
-	/* readahead works cnt */
-	atomic_t reada_works_cnt;
-
 	/* Extent buffer radix tree */
 	spinlock_t buffer_lock;
 	/* Entries are eb->start / sectorsize */
@@ -3836,23 +3828,6 @@ static inline void btrfs_bio_counter_dec(struct btrfs_fs_info *fs_info)
 	btrfs_bio_counter_sub(fs_info, 1);
 }
 
-/* reada.c */
-struct reada_control {
-	struct btrfs_fs_info	*fs_info;		/* tree to prefetch */
-	struct btrfs_key	key_start;
-	struct btrfs_key	key_end;	/* exclusive */
-	atomic_t		elems;
-	struct kref		refcnt;
-	wait_queue_head_t	wait;
-};
-struct reada_control *btrfs_reada_add(struct btrfs_root *root,
-			      struct btrfs_key *start, struct btrfs_key *end);
-int btrfs_reada_wait(void *handle);
-void btrfs_reada_detach(void *handle);
-int btree_readahead_hook(struct extent_buffer *eb, int err);
-void btrfs_reada_remove_dev(struct btrfs_device *dev);
-void btrfs_reada_undo_remove_dev(struct btrfs_device *dev);
-
 static inline int is_fstree(u64 rootid)
 {
 	if (rootid == BTRFS_FS_TREE_OBJECTID ||
fs/btrfs/dev-replace.c +0 -5
index 66fa61cb3f23..62b9651ea662 100644
@@ -906,9 +906,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 	}
 	btrfs_wait_ordered_roots(fs_info, U64_MAX, 0, (u64)-1);
 
-	if (!scrub_ret)
-		btrfs_reada_remove_dev(src_device);
-
 	/*
 	 * We have to use this loop approach because at this point src_device
 	 * has to be available for transaction commit to complete, yet new
@@ -917,7 +914,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 	while (1) {
 		trans = btrfs_start_transaction(root, 0);
 		if (IS_ERR(trans)) {
-			btrfs_reada_undo_remove_dev(src_device);
 			mutex_unlock(&dev_replace->lock_finishing_cancel_unmount);
 			return PTR_ERR(trans);
 		}
@@ -968,7 +964,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 		up_write(&dev_replace->rwsem);
 		mutex_unlock(&fs_info->chunk_mutex);
 		mutex_unlock(&fs_info->fs_devices->device_list_mutex);
-		btrfs_reada_undo_remove_dev(src_device);
 		btrfs_rm_dev_replace_blocked(fs_info);
 		if (tgt_device)
 			btrfs_destroy_dev_replace_tgtdev(tgt_device);
fs/btrfs/disk-io.c +2 -18
index 5c598e124c25..8d7438aecdd0 100644
@@ -665,9 +665,6 @@ static int validate_subpage_buffer(struct page *page, u64 start, u64 end,
 	if (ret < 0)
 		goto err;
 
-	if (test_and_clear_bit(EXTENT_BUFFER_READAHEAD, &eb->bflags))
-		btree_readahead_hook(eb, ret);
-
 	set_extent_buffer_uptodate(eb);
 
 	free_extent_buffer(eb);
@@ -715,10 +712,6 @@ int btrfs_validate_metadata_buffer(struct btrfs_bio *bbio,
 	}
 	ret = validate_extent_buffer(eb);
 err:
-	if (reads_done &&
-	    test_and_clear_bit(EXTENT_BUFFER_READAHEAD, &eb->bflags))
-		btree_readahead_hook(eb, ret);
-
 	if (ret) {
 		/*
 		 * our io error hook is going to dec the io pages
@@ -2224,7 +2217,6 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
 	btrfs_destroy_workqueue(fs_info->endio_freespace_worker);
 	btrfs_destroy_workqueue(fs_info->delayed_workers);
 	btrfs_destroy_workqueue(fs_info->caching_workers);
-	btrfs_destroy_workqueue(fs_info->readahead_workers);
 	btrfs_destroy_workqueue(fs_info->flush_workers);
 	btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
 	if (fs_info->discard_ctl.discard_workers)
@@ -2437,9 +2429,6 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info)
 	fs_info->delayed_workers =
 		btrfs_alloc_workqueue(fs_info, "delayed-meta", flags,
 				      max_active, 0);
-	fs_info->readahead_workers =
-		btrfs_alloc_workqueue(fs_info, "readahead", flags,
-				      max_active, 2);
 	fs_info->qgroup_rescan_workers =
 		btrfs_alloc_workqueue(fs_info, "qgroup-rescan", flags, 1, 0);
 	fs_info->discard_ctl.discard_workers =
@@ -2451,9 +2440,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info)
 	      fs_info->endio_meta_write_workers &&
 	      fs_info->endio_write_workers && fs_info->endio_raid56_workers &&
 	      fs_info->endio_freespace_worker && fs_info->rmw_workers &&
-	      fs_info->caching_workers && fs_info->readahead_workers &&
-	      fs_info->fixup_workers && fs_info->delayed_workers &&
-	      fs_info->qgroup_rescan_workers &&
+	      fs_info->caching_workers && fs_info->fixup_workers &&
+	      fs_info->delayed_workers && fs_info->qgroup_rescan_workers &&
 	      fs_info->discard_ctl.discard_workers)) {
 		return -ENOMEM;
 	}
@@ -3083,7 +3071,6 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)
 
 	atomic_set(&fs_info->async_delalloc_pages, 0);
 	atomic_set(&fs_info->defrag_running, 0);
-	atomic_set(&fs_info->reada_works_cnt, 0);
 	atomic_set(&fs_info->nr_delayed_iputs, 0);
 	atomic64_set(&fs_info->tree_mod_seq, 0);
 	fs_info->global_root_tree = RB_ROOT;
@@ -3094,9 +3081,6 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)
 	fs_info->tree_mod_log = RB_ROOT;
 	fs_info->commit_interval = BTRFS_DEFAULT_COMMIT_INTERVAL;
 	fs_info->avg_delayed_ref_runtime = NSEC_PER_SEC >> 6; /* div by 64 */
-	/* readahead state */
-	INIT_RADIX_TREE(&fs_info->reada_tree, GFP_NOFS & ~__GFP_DIRECT_RECLAIM);
-	spin_lock_init(&fs_info->reada_lock);
 	btrfs_init_ref_verify(fs_info);
 
 	fs_info->thread_pool_size = min_t(unsigned long,
fs/btrfs/extent_io.c +0 -3
index 1a67f4b3986b..c9f1f5f3846b 100644
@@ -3087,9 +3087,6 @@ static void end_bio_extent_readpage(struct bio *bio)
 			set_bit(EXTENT_BUFFER_READ_ERR, &eb->bflags);
 			eb->read_mirror = mirror;
 			atomic_dec(&eb->io_pages);
-			if (test_and_clear_bit(EXTENT_BUFFER_READAHEAD,
-					       &eb->bflags))
-				btree_readahead_hook(eb, -EIO);
 		}
 readpage_ok:
 		if (likely(uptodate)) {
fs/btrfs/reada.c deleted +0 -1086

1,086 lines not on this page · raw message

fs/btrfs/scrub.c +0 -36
index 70457aadd721..5e59e27e6bcd 100644
@@ -3195,10 +3195,7 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	u64 physical_end;
 	u64 generation;
 	int mirror_num;
-	struct reada_control *reada1;
-	struct reada_control *reada2;
 	struct btrfs_key key;
-	struct btrfs_key key_end;
 	u64 increment = map->stripe_len;
 	u64 offset;
 	u64 extent_logical;
@@ -3249,11 +3246,6 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	path->skip_locking = 1;
 	path->reada = READA_FORWARD;
 
-	/*
-	 * trigger the readahead for extent tree csum tree and wait for
-	 * completion. During readahead, the scrub is officially paused
-	 * to not hold off transaction commits
-	 */
 	logical = base + offset;
 	physical_end = physical + nstripes * map->stripe_len;
 	if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
@@ -3268,36 +3260,8 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
 	scrub_blocked_if_needed(fs_info);
 
 	root = btrfs_extent_root(fs_info, logical);
-
-	/* FIXME it might be better to start readahead at commit root */
-	key.objectid = logical;
-	key.type = BTRFS_EXTENT_ITEM_KEY;
-	key.offset = (u64)0;
-	key_end.objectid = logic_end;
-	key_end.type = BTRFS_METADATA_ITEM_KEY;
-	key_end.offset = (u64)-1;
-	reada1 = btrfs_reada_add(root, &key, &key_end);
-
 	csum_root = btrfs_csum_root(fs_info, logical);
 
-	if (cache->flags & BTRFS_BLOCK_GROUP_DATA) {
-		key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
-		key.type = BTRFS_EXTENT_CSUM_KEY;
-		key.offset = logical;
-		key_end.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
-		key_end.type = BTRFS_EXTENT_CSUM_KEY;
-		key_end.offset = logic_end;
-		reada2 = btrfs_reada_add(csum_root, &key, &key_end);
-	} else {
-		reada2 = NULL;
-	}
-
-	if (!IS_ERR(reada1))
-		btrfs_reada_wait(reada1);
-	if (!IS_ERR_OR_NULL(reada2))
-		btrfs_reada_wait(reada2);
-
-
 	/*
 	 * collect all data csums for the stripe to avoid seeking during
 	 * the scrub. This might currently (crc32) end up to be about 1MB
fs/btrfs/super.c +0 -1
index a1c54a2c787c..0ec09fe01be6 100644
@@ -1842,7 +1842,6 @@ static void btrfs_resize_thread_pool(struct btrfs_fs_info *fs_info,
 	btrfs_workqueue_set_max(fs_info->endio_write_workers, new_pool_size);
 	btrfs_workqueue_set_max(fs_info->endio_freespace_worker, new_pool_size);
 	btrfs_workqueue_set_max(fs_info->delayed_workers, new_pool_size);
-	btrfs_workqueue_set_max(fs_info->readahead_workers, new_pool_size);
 	btrfs_workqueue_set_max(fs_info->scrub_wr_completion_workers,
 				new_pool_size);
 }
fs/btrfs/volumes.c +0 -7
index f38c230111be..141fd7b4b018 100644
@@ -1166,7 +1166,6 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 	ASSERT(!test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state));
 	ASSERT(list_empty(&device->dev_alloc_list));
 	ASSERT(list_empty(&device->post_commit_list));
-	ASSERT(atomic_read(&device->reada_in_flight) == 0);
 }
 
 static void close_fs_devices(struct btrfs_fs_devices *fs_devices)
@@ -2148,8 +2147,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info,
 	}
 
 	ret = btrfs_shrink_device(device, 0);
-	if (!ret)
-		btrfs_reada_remove_dev(device);
 	if (ret)
 		goto error_undo;
 
@@ -2247,7 +2244,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info,
 	return ret;
 
 error_undo:
-	btrfs_reada_undo_remove_dev(device);
 	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state)) {
 		mutex_lock(&fs_info->chunk_mutex);
 		list_add(&device->dev_alloc_list,
@@ -6978,11 +6974,8 @@ struct btrfs_device *btrfs_alloc_device(struct btrfs_fs_info *fs_info,
 	INIT_LIST_HEAD(&dev->dev_alloc_list);
 	INIT_LIST_HEAD(&dev->post_commit_list);
 
-	atomic_set(&dev->reada_in_flight, 0);
 	atomic_set(&dev->dev_stats_ccnt, 0);
 	btrfs_device_data_ordered_init(dev);
-	INIT_RADIX_TREE(&dev->reada_zones, GFP_NOFS & ~__GFP_DIRECT_RECLAIM);
-	INIT_RADIX_TREE(&dev->reada_extents, GFP_NOFS & ~__GFP_DIRECT_RECLAIM);
 	extent_io_tree_init(fs_info, &dev->alloc_state,
 			    IO_TREE_DEVICE_ALLOC_STATE, NULL);
 
fs/btrfs/volumes.h +0 -7
index 3b8130680749..b2467a9159d0 100644
@@ -123,13 +123,6 @@ struct btrfs_device {
 	/* per-device scrub information */
 	struct scrub_ctx *scrub_ctx;
 
-	/* readahead state */
-	atomic_t reada_in_flight;
-	u64 reada_next;
-	struct reada_zone *reada_curr_zone;
-	struct radix_tree_root reada_zones;
-	struct radix_tree_root reada_extents;
-
 	/* disk I/O failure stats. For detailed description refer to
 	 * enum btrfs_dev_stat_values in ioctl.h */
 	int dev_stats_valid;
Signature · 1 line
2.34.1
#4 lore raw
In reply to Qu Wenruo
Quoted 111 lines · Qu Wenruo

[PROBLEMS]
The metadata readahead code is introduced in 2011 (surprisingly, the commit message even contains changelog), but now there is only one user for it, and even for the only one user, the readahead mechanism can't provide all it needs:

- No support for commit tree readahead
Only support readahead on current tree.

- Bad layer separation
To manage on-fly bios, btrfs_reada_add() mechanism internally manages a kinda complex zone system, and it's bind to per-device.

This is against the common layer separation, as metadata should all be in btrfs logical address space, should not be bound to device physical layer.

In fact, this is the cause of all recent reada related bugs.

- No caller for asynchronous metadata readahead
Even btrfs_reada_add() is designed to be fully asynchronous, scrub only calls it in a synchronous way (call btrfs_reada_add() and immediately call btrfs_reada_wait()).
Thus rendering a lot of code unnecessary.

[ALTERNATIVE]
On the other hand, we have btrfs_path::reada mechanism, which is already used by tons of btrfs sub-systems like send.

[MODIFICATION]
This patch will use btrfs_path::reada to replace btrfs_reada_add() mechanism.

[BENCHMARK]
The conclusion looks like this:

For the worst case (no dirty metadata, slow HDD), there will be around 5% performance drop for scrub.
For other cases (even SATA SSD), there is no distinguishable performance difference.

The number is reported scrub speed, in MiB/s.
The resolution is limited by the reported duration, which only has a resolution of 1 second.

	Old		New		Diff

SSD 455.3 466.332 +2.42%
HDD 103.927 98.012 -5.69%

[BENCHMARK DETAILS]
Both tests are done in the same host and VM, the VM has one dedicated SSD and one dedicated HDD attached to it (virtio driver though)

Host:
CPU: 5900X
RAM: DDR4 3200MT, no ECC

During the benchmark, there is no other active other than light DE operations.

VM:
vCPU: 16x
RAM: 4G

The VM kernel doesn't have any heavy debug options to screw up the benchmark result.

Test drives:
SSD: 500G SATA SSD

	Crucial CT500MX500SSD1
	(With quite some wear, as it's my main test drive for fstests)

HDD: 2T 5400rpm SATA HDD (device-managed SMR)

	WDC WD20SPZX-22UA7T0
	(Very new, invested just for this benchmark)

Filesystem contents:
For filesystems on SSD and HDD, they are generated using the following fio job file:

[scrub-populate]
directory=/mnt/btrfs
nrfiles=16384
openfiles=16
filesize=2k-512k
readwrite=randwrite
ioengine=libaio
fallocate=none
numjobs=4

Then randomly remove 4096 files (1/16th of total files) to create enough gaps in extent tree.

Finally run scrub on each filesystem 5 times, with cycle mount and module reload between each run.

Full result can be fetched here:
https://docs.google.com/spreadsheets/d/1cwUAlbKPfp4baKrS92debsCt6Ejqvxr_Ylspj_SDFT0/edit?usp=sharing

[CHANGELOG]
RFC->v1:
- Add benchmark result
- Add extent tree readahead using btrfs_path::reada

v2:
- Reorder the patches
So that the reada removal comes at last

- Add benchmark result into the reada removal patch

Quoted 7 lines

- Fix a bug that can cause false alert for RAID56 dev-replace/scrub Caused by missing ->search_commit_root assignment during refactor.

Qu Wenruo (3):
btrfs: remove the unnecessary path parameter for scrub_raid56_parity()
btrfs: use btrfs_path::reada for scrub extent tree readahead
btrfs: remove reada mechanism

Added to misc-next, thanks.