Skip to content

btrfs: scrub: rework to get rid of the complex bio formshaping

Qu Wenruo linux-btrfs llvm oe-kbuild-all 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest11 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11btrfs: scrub: introduce the structure for new BTRFS_STRIPE_LEN based interfacefs/btrfs/scrub.c fs/btrfs/scrub.h+162 -00 replies
2/11btrfs: scrub: introduce a helper to find and fill the sector info for a scrub2_stripefs/btrfs/file-item.c fs/btrfs/file-item.h fs/btrfs/raid56.c +2 more+148 -30 replies
3/11btrfs: scrub: introduce a helper to verify one scrub2_stripefs/btrfs/disk-io.c fs/btrfs/disk-io.h fs/btrfs/scrub.c +1 more+161 -50 replies
4/11btrfs: scrub: add the repair function for scrub2_stripefs/btrfs/scrub.c fs/btrfs/scrub.h+158 -20 replies
5/11btrfs: scrub: add a writeback helper for scrub2_stripefs/btrfs/scrub.c fs/btrfs/scrub.h+111 -01 reply
6/11btrfs: scrub: add the error reporting for scrub2_stripefs/btrfs/scrub.c fs/btrfs/scrub.h+158 -90 replies
7/11btrfs: scrub: add raid56 P/Q scrubbing supportfs/btrfs/scrub.c fs/btrfs/scrub.h+257 -60 replies
8/11btrfs: scrub: use dedicated super block verification function to scrub one super blockfs/btrfs/scrub.c+39 -40 replies
9/11btrfs: scrub: switch to the new scrub2_stripe based infrastructurefs/btrfs/scrub.c fs/btrfs/scrub.h+98 -1150 replies
10/11btrfs: scrub: cleanup the old scrub_parity infrastructurefs/btrfs/scrub.c fs/btrfs/scrub.h+0 -5080 replies
11/11btrfs: scrub: cleanup scrub_extent() and its related functionsfs/btrfs/scrub.c fs/btrfs/scrub.h+0 -4520 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PoC PATCH 00/11] btrfs: scrub: rework to get rid of the complex bio formshaping

This is a proof-of-concept patchset, thus don't merge.

This series is mostly a full rework of low level scrub code.

Although I implemented the full support for all profiles, only raid56 code is partially cleaned up (which is already over 1K lines removed). The estimated full cleanup will be around 1~2K more lines removed eventually.

The series is sent out for feedback, as the full patchset can be very large (mostly to remove old codes).

[PROBLEMS OF SCRUB]

TL;DR
The current scrub code is way too complex for future expansion.

Current scrub code has a complex system to manage its in-flight bios.

This behavior is designed to improve scrub performance, but has a lot of disadvantage too:

- Too many indirect calls/jumps

To scrub one extent in a simple mirror (like SINGLE), the call chain involves the following functions:

  /* Before entering scrub_simple_mirror() */
  scrub_ctx()
  |- INIT_WORK(&sbio->work, scrub_bio_end_io_worker);

  /* In scrub_simple_mirror() */
  scrub_extent()
  |- scrub_sectors()
     |- scrub_add_sector_to_rd_bio()
        |- sbio->bio->bi_end_io = scrub_bio_end_io;

/* Now it jumps to the endio function */

scrub_bio_end_io()
|- queue_work()

/* Now it jumps to workqueue, which is setup in scrub_ctx() */ scrub_bio_end_io_worker()
|- scrub_block_complete()

     |- scrub_handle_errored_block() /* For corruption case */
     |  |- scrub_recheck_block()
     |     |- scrub_repair_block_from_good_copy()
     |- scrub_checksum()
     |- scrub_write_block_to_dev_replace()

The whole jumps/delayed calls are really a mess to read. This makes me wonder if the original code is really designed for human to read.

- Complex bio form-shaping
We have scrub_bio to manage the in-flight bios.

It has at most 64 bios for one device scrub, and each bio can be as large as 128K.

This is definitely a big performance enhancement.

But I'm not convinced that scrub performance should be the first thing to consider.

- No usage of btrfs_submit_bio()
This means we're doing a lot of things which can already be handled by btrfs_submit_bio().

This means quite some duplicated code.

[ENHANCEMENT]

This patchset will introduce a new infrasturcture, scrub2_stripe.

The "scrub2" prefix is to avoid naming conflicts.

The overall idea is, we always do scrub in the unit of BTRFS_STRIPE_LEN, hence the "scrub2_stripe".

Furthermore, all work will done in a submit-and-wait fashion, reducing the delayed calls/jumps to minimal.

The new scrub entrance in scrub_simple_mirror() would looks like this:

scrub_simple_mirror()

  |  /* Find a stripe with at least one sector used */
  |- scrub2_find_fill_first_stripe()
  |
  |  /* Submit a read for the whole 64KiB */
  |- scrub2_read_and_wait_stripe()
  |  |- btrfs_submit_bio()
  |  |  /* do the verification for all sectors */
  |  |- scrub2_verify_one_stripe()
  |
  |- scrub2_reapair_one_stripe()
  |  |- scrub2_repair_from_mirror()
  |     |  /* reuse the existing read path */
  |     |- scrub2_read_and_wait_stripe()
  |
  |- scrub2_report_errors()
  |
  |  /*
  |   * For dev-replace and regular scrub repair, the write range
  |   * will be different.
  |   * (replace will writeback all good sectors, repair only writes
  |   *  back repaired sectors).
  |   */
  |- scrub2_writeback_sectors()

Everything is done in a submit-and-wait fashion.

Although this will reduce concurrency, the readability will be greatly improved.

Furthermore since we're already submitting read in 64KiB size, it's less IOPS intense, thus it's already doing optimization for read.

Even if the performance is not that good, it can be enhanced later by using scrub2_stripe_group to submit multiple stripes in one go (aka, enlarge the block size) to improve performance, without damaging readability.

[CURRENT FEATURES]

- Working scrub/replace for all profiles
Currently only non-raid56 is tested.

[TODO]

There are a lot of todos:

- Completely remove scrub_bio/scrub_block/scrub_sector I estimate that would result further 1~2K lines removed.

Unfortunately, thus huge cleanup will take way more patches than usual.

- Add proper support for zoned devices
Currently zoned code is not touched, but existing zoned code relies on scrub_bio.
Thus they won't work at all.

- Proper performance benchmarking

Qu Wenruo (11):
btrfs: scrub: introduce the structure for new BTRFS_STRIPE_LEN based
interface
btrfs: scrub: introduce a helper to find and fill the sector info for
a scrub2_stripe
btrfs: scrub: introduce a helper to verify one scrub2_stripe
btrfs: scrub: add the repair function for scrub2_stripe
btrfs: scrub: add a writeback helper for scrub2_stripe
btrfs: scrub: add the error reporting for scrub2_stripe
btrfs: scrub: add raid56 P/Q scrubbing support
btrfs: scrub: use dedicated super block verification function to scrub
one super block
btrfs: scrub: switch to the new scrub2_stripe based infrastructure
btrfs: scrub: cleanup the old scrub_parity infrastructure
btrfs: scrub: cleanup scrub_extent() and its related functions

 fs/btrfs/disk-io.c   |   10 +-
 fs/btrfs/disk-io.h   |    2 +
 fs/btrfs/file-item.c |    8 +-
 fs/btrfs/file-item.h |    3 +-
 fs/btrfs/raid56.c    |    2 +-
 fs/btrfs/scrub.c     | 2263 +++++++++++++++++++++++-------------------
 6 files changed, 1238 insertions(+), 1050 deletions(-)
Signature · 1 line
2.38.1