Skip to contentSkip to conversation

btrfs: simplify the stripe buffer handling

PATCH Superseded Qu Wenruo 3 participants 5 messages Jun 29 – Jun 30 Btrfs file system

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v3 cover btrfs: simplify the stripe buffer handling

[CHANGELOG]
v3:
- Add a patch to remove a unused macro

- Add new ASSERT()s to make sure the kvmallocated memory is page aligned The size is 64K, which is page aligned and power of 2, thus the resulted memory should at least be page aligned.

Furthermore, we have several call sites using (BTRFS_STRIPE_LEN / PAGE_SIZE) or (BTRFS_STRIPE_LEN >> PAGE_SHIFT), add an build time assert to make sure our PAGE_SIZE is not larger than BTRFS_STRIPE_LEN.

v2:
- Fix a potential double-free during release_scrub_stripe() Reported by sashiko.

- Enlarge the scrub bbio to be able to contain the full buffer Since the buffer is allocated by kvmalloc(), it's possible that not all pages are physically contiguous.

In that case we have to enlarge the bbio to contain (BTRFS_STRIPE_LEN >> PAGE_SHIFT) bvecs.

This is to use kvmalloc() to allocate a buffer for scrub, so that we do not need to bother page-switching, and simplify the scrub progress.

Qu Wenruo (2):
btrfs: remove SCRUB_MAX_SECTORS_PER_BLOCK
btrfs: use kvmalloc() for stripe buffer of scrub_stripe

 fs/btrfs/raid56.c |  18 +----
 fs/btrfs/raid56.h |   2 +-
 fs/btrfs/scrub.c  | 183 ++++++++++++++++++++--------------------------
 3 files changed, 85 insertions(+), 118 deletions(-)
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH v3 2/2 btrfs: use kvmalloc() for stripe buffer of scrub_stripe

Currently we're using scrub_stripe::folios[] to store all contents of a stripe.

This means we need all the extra work to handle things like sub-page cases, and also require larger folios to handle bs > ps cases.

On the other hand, it's not hard to allocate a 64K large folio to cover the full stripe, getting rid of the cross-page handling.

Furthermore, even if that large folio allocation failed, we can still use vmalloc() to allocate a virtually contiguous space and still get rid of cross-page handling.

This patch will go with kvmalloc() to allocate 64K of memory for the stripe buffer, thus getting rid of all the complex cross-page handling.

The following aspects can be greatly simplified:

- Checksum verification for both data and metadata No more per-page iteration, all in one go.

- RAID56 data caching
Just copy the buffer into the RAID56 pages.

- No more kaddr/paddr grabbing
For most cases the virtual address is enough for csum calculation and io submission.
There is only one location left with a kernel address requirement, that's calc_sector_number, which needs to compare against the bvec virtual address.

Just open-code a simple vaddr->kaddr converter for that only caller.

- Bio assembly
There is already the helper bio_add_vmalloc() to queue vmallocated memory into a bio.

Although it means we have something else to be concerned about:

- Bio assembly
If the memory is vmallocated, we need to use bio_add_vmalloc() Otherwise use the existing bio_add_page().

- Read endio
For vmallocated memory, we need to call
invalidate_kernel_vmap_range().

- Scrub bbio bvec size
Since scrub_stripe::buffer is kvmallocated, we also need to enlarge the scrub bbio, to be able to handle the worst case, where all 64KiB is allocated by discontiguous 4K physical pages.

---
 fs/btrfs/raid56.c |  18 ++---
 fs/btrfs/raid56.h |   2 +-
 fs/btrfs/scrub.c  | 169 ++++++++++++++++++++++------------------------
 3 files changed, 85 insertions(+), 104 deletions(-)
fs/btrfs/raid56.c +4 -14
index f7f7db40994c..0faff694eb78 100644
@@ -2996,13 +2996,11 @@ void raid56_parity_submit_scrub_rbio(struct btrfs_raid_bio *rbio)
  * This is due to the fact rbio has its own page management for its cache.
  */
 void raid56_parity_cache_data_folios(struct btrfs_raid_bio *rbio,
-				     struct folio **data_folios, u64 data_logical)
+				     void *vaddr, u64 data_logical)
 {
 	struct btrfs_fs_info *fs_info = rbio->bioc->fs_info;
 	const u64 offset_in_full_stripe = data_logical -
 					  rbio->bioc->full_stripe_logical;
-	unsigned int findex = 0;
-	unsigned int foffset = 0;
 	int ret;
 
 	/*
@@ -3025,18 +3023,10 @@ void raid56_parity_cache_data_folios(struct btrfs_raid_bio *rbio,
 	     cur_off < offset_in_full_stripe + BTRFS_STRIPE_LEN;
 	     cur_off += PAGE_SIZE) {
 		const unsigned int pindex = cur_off >> PAGE_SHIFT;
-		void *kaddr;
 
-		kaddr = kmap_local_page(rbio->stripe_pages[pindex]);
-		memcpy_from_folio(kaddr, data_folios[findex], foffset, PAGE_SIZE);
-		kunmap_local(kaddr);
-
-		foffset += PAGE_SIZE;
-		ASSERT(foffset <= folio_size(data_folios[findex]));
-		if (foffset == folio_size(data_folios[findex])) {
-			findex++;
-			foffset = 0;
-		}
+		ASSERT(cur_off - offset_in_full_stripe + PAGE_SIZE <= BTRFS_STRIPE_LEN);
+		memcpy_to_page(rbio->stripe_pages[pindex], 0,
+			       vaddr + cur_off - offset_in_full_stripe, PAGE_SIZE);
 	}
 	bitmap_set(rbio->stripe_uptodate_bitmap,
 		   offset_in_full_stripe >> fs_info->sectorsize_bits,
fs/btrfs/raid56.h +1 -1
index 1f463ecf7e41..8542648199f1 100644
@@ -283,7 +283,7 @@ struct btrfs_raid_bio *raid56_parity_alloc_scrub_rbio(struct bio *bio,
 void raid56_parity_submit_scrub_rbio(struct btrfs_raid_bio *rbio);
 
 void raid56_parity_cache_data_folios(struct btrfs_raid_bio *rbio,
-				     struct folio **data_folios, u64 data_logical);
+				     void *vaddr, u64 data_logical);
 
 int btrfs_alloc_stripe_hash_table(struct btrfs_fs_info *info);
 void btrfs_free_stripe_hash_table(struct btrfs_fs_info *info);
fs/btrfs/scrub.c +80 -89

331 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#1 lore raw

PATCH v3 1/2 btrfs: remove SCRUB_MAX_SECTORS_PER_BLOCK

In reply to Qu Wenruo

The last user of this macro is removed in commit 001e3fc263ce ("btrfs:
scrub: remove scrub_block and scrub_sector structures").

Now that macro is only utilized in an ASSERT(), which no longer makes much sense.

Just remove it completely.

---
 fs/btrfs/scrub.c | 14 --------------
 1 file changed, 14 deletions(-)
fs/btrfs/scrub.c +0 -14
index d2f7ac5b6e96..5ec04c91e12d 100644
@@ -57,12 +57,6 @@ struct scrub_ctx;
 
 #define SCRUB_TOTAL_STRIPES		(SCRUB_GROUPS_PER_SCTX * SCRUB_STRIPES_PER_GROUP)
 
-/*
- * The following value times PAGE_SIZE needs to be large enough to match the
- * largest node/leaf/sector size that shall be supported.
- */
-#define SCRUB_MAX_SECTORS_PER_BLOCK	(BTRFS_MAX_METADATA_BLOCKSIZE / SZ_4K)
-
 /* Represent one sector and its needed info to verify the content. */
 struct scrub_sector_verification {
 	union {
@@ -3091,14 +3085,6 @@ int btrfs_scrub_dev(struct btrfs_fs_info *fs_info, u64 devid, u64 start,
 	/* At mount time we have ensured nodesize is in the range of [4K, 64K]. */
 	ASSERT(fs_info->nodesize <= BTRFS_STRIPE_LEN);
 
-	/*
-	 * SCRUB_MAX_SECTORS_PER_BLOCK is calculated using the largest possible
-	 * value (max nodesize / min sectorsize), thus nodesize should always
-	 * be fine.
-	 */
-	ASSERT(fs_info->nodesize <=
-	       SCRUB_MAX_SECTORS_PER_BLOCK << fs_info->sectorsize_bits);
-
 	/* Allocate outside of device_list_mutex */
 	sctx = scrub_setup_ctx(fs_info, is_dev_replace);
 	if (IS_ERR(sctx))
Signature · 1 line
2.54.0
#3 lore raw

Re: [PATCH v3 2/2] btrfs: use kvmalloc() for stripe buffer of scrub_stripe

In reply to Qu Wenruo
Quoted 13 lines · Qu Wenruo

Currently we're using scrub_stripe::folios[] to store all contents of a stripe.

This means we need all the extra work to handle things like sub-page cases, and also require larger folios to handle bs > ps cases.

On the other hand, it's not hard to allocate a 64K large folio to cover the full stripe, getting rid of the cross-page handling.

Furthermore, even if that large folio allocation failed, we can still use vmalloc() to allocate a virtually contiguous space and still get rid of cross-page handling.

Quoted 5 lines

This patch will go with kvmalloc() to allocate 64K of memory for the stripe buffer, thus getting rid of all the complex cross-page handling.

The following aspects can be greatly simplified:

I think the trade off using the virtual mappings is justified, the code is indeed simplified.

fs/btrfs/scrub.c static void release_scrub_stripe(struct scrub_stripe *stripe) · line 336

Quoted 10 lines
fs/btrfs/scrub.c +1 -5
@@ -332,13 +333,10 @@ static void release_scrub_stripe(struct scrub_stripe *stripe)
 	if (!stripe)
 		return;
 
-	for (int i = 0; i < SCRUB_STRIPE_MAX_FOLIOS; i++) {
-		if (stripe->folios[i])
-			folio_put(stripe->folios[i]);
-		stripe->folios[i] = NULL;
-	}
+	kvfree(stripe->buffer);

The life time of the stripe->buffer matches the scrub life time so we won't be allocating and freeing the page repeatedly.

Quoted 8 lines
fs/btrfs/scrub.c +1 -0
 	kfree(stripe->sectors);
 	kfree(stripe->csums);
+	stripe->buffer = NULL;
 	stripe->sectors = NULL;
 	stripe->csums = NULL;
 	stripe->sctx = NULL;

fs/btrfs/scrub.c static void scrub_verify_one_sector(struct scrub_stripe *stripe, int sector_nr) · line 811

Quoted 9 lines
fs/btrfs/scrub.c +4 -2
@@ -840,8 +805,10 @@ static void scrub_verify_one_sector(struct scrub_stripe *stripe, int sector_nr)
 		return;
 	}
 
-	ret = btrfs_check_block_csum(fs_info, paddr, csum_buf, sector->csum);
-	if (ret < 0) {
+	scrub_calc_vaddr_csum(fs_info,
+			      stripe->buffer + (sector_nr << fs_info->sectorsize_bits),
+			      fs_info->sectorsize, csum_buf);
+	if (memcmp(csum_buf, sector->csum, fs_info->csum_size)) {

Please use the if (memcmp(...) != 0) form

fs/btrfs/scrub.c static void scrub_repair_read_endio(struct btrfs_bio *bbio) · line 874

Quoted 8 lines
fs/btrfs/scrub.c +4 -0
@@ -891,6 +868,10 @@ static void scrub_repair_read_endio(struct btrfs_bio *bbio)
 
 	ASSERT(sector_nr < stripe->nr_sectors);
 
+	if (is_vmalloc_addr(stripe->buffer))
+		invalidate_kernel_vmap_range(
+			stripe->buffer + (sector_nr << fs_info->sectorsize_bits),
+			bio_size);

This is done twice so may not require a helper but it's quite important for the virtual mappings, so a comment making it explicit that read side must do the invalidation would be good.

Alternatively I was thinking about naming like "btrfs_invalidate_range_before_read" but a comment would be more clear, also we'd need to pass 2 more variables.

Otherwise, I went through this patch a few times and haven't spotted anything suspicious.

#4 lore raw

Re: [PATCH v3 2/2] btrfs: use kvmalloc() for stripe buffer of scrub_stripe

Quoted 61 lines · David Sterba
Quoted 13 lines · Qu Wenruo

Currently we're using scrub_stripe::folios[] to store all contents of a stripe.

This means we need all the extra work to handle things like sub-page cases, and also require larger folios to handle bs > ps cases.

On the other hand, it's not hard to allocate a 64K large folio to cover the full stripe, getting rid of the cross-page handling.

Furthermore, even if that large folio allocation failed, we can still use vmalloc() to allocate a virtually contiguous space and still get rid of cross-page handling.

Quoted 5 lines

This patch will go with kvmalloc() to allocate 64K of memory for the stripe buffer, thus getting rid of all the complex cross-page handling.

The following aspects can be greatly simplified:

I think the trade off using the virtual mappings is justified, the code is indeed simplified.

fs/btrfs/scrub.c static void release_scrub_stripe(struct scrub_stripe *stripe) · line 336

Quoted 10 lines
fs/btrfs/scrub.c +1 -5
@@ -332,13 +333,10 @@ static void release_scrub_stripe(struct scrub_stripe *stripe)
  	if (!stripe)
  		return;
 
-	for (int i = 0; i < SCRUB_STRIPE_MAX_FOLIOS; i++) {
-		if (stripe->folios[i])
-			folio_put(stripe->folios[i]);
-		stripe->folios[i] = NULL;
-	}
+	kvfree(stripe->buffer);

The life time of the stripe->buffer matches the scrub life time so we won't be allocating and freeing the page repeatedly.

Quoted 8 lines
fs/btrfs/scrub.c +1 -0
  	kfree(stripe->sectors);
  	kfree(stripe->csums);
+	stripe->buffer = NULL;
  	stripe->sectors = NULL;
  	stripe->csums = NULL;
  	stripe->sctx = NULL;

fs/btrfs/scrub.c static void scrub_verify_one_sector(struct scrub_stripe *stripe, int sector_nr) · line 811

Quoted 9 lines
fs/btrfs/scrub.c +4 -2
@@ -840,8 +805,10 @@ static void scrub_verify_one_sector(struct scrub_stripe *stripe, int sector_nr)
  		return;
  	}
 
-	ret = btrfs_check_block_csum(fs_info, paddr, csum_buf, sector->csum);
-	if (ret < 0) {
+	scrub_calc_vaddr_csum(fs_info,
+			      stripe->buffer + (sector_nr << fs_info->sectorsize_bits),
+			      fs_info->sectorsize, csum_buf);
+	if (memcmp(csum_buf, sector->csum, fs_info->csum_size)) {

Please use the if (memcmp(...) != 0) form

Quoted 4 lines
fs/btrfs/scrub.c +0 -0
@@ -891,6 +868,10 @@ static void scrub_repair_read_endio(struct btrfs_bio *bbio)
 
  	ASSERT(sector_nr < stripe->nr_sectors);
 
Quoted 8 lines

fs/btrfs/scrub.c static void scrub_repair_read_endio(struct btrfs_bio *bbio) · line 874

Quoted 4 lines
fs/btrfs/scrub.c +4 -0
+	if (is_vmalloc_addr(stripe->buffer))
+		invalidate_kernel_vmap_range(
+			stripe->buffer + (sector_nr << fs_info->sectorsize_bits),
+			bio_size);

This is done twice so may not require a helper but it's quite important for the virtual mappings, so a comment making it explicit that read side must do the invalidation would be good.

I can refactor the code so that scrub_repair_read_endio() and scrub_read_endio() share a common function, only diff handling after the bio is put.

In that case, we can easily add a common for the shared path code.

Thanks,
Qu

Quoted 8 lines

Alternatively I was thinking about naming like "btrfs_invalidate_range_before_read" but a comment would be more clear, also we'd need to pass 2 more variables.

Otherwise, I went through this patch a few times and haven't spotted anything suspicious.