Skip to content

Big rebalance changes

Kent Overstreet linux-bcachefs 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest21 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/21bcachefs: s/bch_io_opts/bch_inode_optsfs/bcachefs/checksum.h fs/bcachefs/data_update.c fs/bcachefs/data_update.h +20 more+75 -750 replies
2/21bcachefs: Inode opt helper refactoringfs/bcachefs/fs-io-buffered.c fs/bcachefs/fs-io-direct.c fs/bcachefs/fs-io.c +6 more+35 -320 replies
3/21bcachefs: opt_change_cookiefs/bcachefs/bcachefs.h fs/bcachefs/inode.c fs/bcachefs/opts.c +2 more+14 -00 replies
4/21bcachefs: Transactional consistency for set_needs_rebalancefs/bcachefs/data_update.c fs/bcachefs/inode.c fs/bcachefs/inode.h +3 more+60 -400 replies
5/21bcachefs: Plumb bch_inode_opts.change_cookiefs/bcachefs/data_update.c fs/bcachefs/inode.c fs/bcachefs/io_misc.c +7 more+19 -150 replies
6/21bcachefs: enum set_needs_rebalance_ctxfs/bcachefs/data_update.c fs/bcachefs/io_write.c fs/bcachefs/move.c +2 more+29 -110 replies
7/21bcachefs: do_rebalance_scan() now responsible for indirect extentsfs/bcachefs/move.c fs/bcachefs/rebalance.c+33 -680 replies
8/21bcachefs: Rename, split out bch2_extent_get_io_opts()fs/bcachefs/move.c fs/bcachefs/move.h fs/bcachefs/rebalance.c +1 more+147 -1340 replies
9/21bcachefs: do_rebalance_extent() uses bch2_extent_get_apply_io_opts()fs/bcachefs/rebalance.c+30 -210 replies
10/21bcachefs: Correct propagation of io options to indirect extentsfs/bcachefs/rebalance.c+14 -40 replies
11/21bcachefs: bkey_should_have_rb_opts()fs/bcachefs/rebalance.c+21 -220 replies
12/21bcachefs: bch2_bkey_needs_rebalance()fs/bcachefs/rebalance.c+81 -900 replies
13/21bcachefs: rebalance now supports changing checksum typefs/bcachefs/rebalance.c+25 -60 replies
14/21bcachefs: Consistency checking for bch_extent_rebalance optsfs/bcachefs/rebalance.c fs/bcachefs/sb-errors_format.h+63 -50 replies
15/21bcachefs: check_rebalance_work checks option inconsistencyfs/bcachefs/rebalance.c+15 -10 replies
16/21bcachefs: bch2_bkey_set_needs_rebalance() now takes per_snapshot_io_optsfs/bcachefs/data_update.c fs/bcachefs/io_write.c fs/bcachefs/rebalance.c +1 more+19 -150 replies
17/21bcachefs: bch_extent_rebalance changesfs/bcachefs/buckets.c fs/bcachefs/data_update.c fs/bcachefs/disk_accounting_format.h +4 more+288 -2800 replies
18/21bcachefs: bch2_set_rebalance_needs_scan_device()fs/bcachefs/rebalance.c fs/bcachefs/rebalance.h fs/bcachefs/super.c +1 more+83 -30 replies
19/21bcachefs: next_rebalance_extent() now handles replicas changesfs/bcachefs/rebalance.c+42 -00 replies
20/21bcachefs: rebalance: erasure_code opt change now handledfs/bcachefs/rebalance.c+20 -40 replies
21/21bcachefs: rebalance_work_(hipri|pending) btreesfs/bcachefs/bcachefs_format.h fs/bcachefs/buckets.c fs/bcachefs/extents.c +3 more+129 -580 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/21] Big rebalance changes

This is a WIP patch series: there's still lots of unfinished bits, but there's big user visible changes happening so it's time to give people a heads up and start circulating for comments and ideas.

This series originated with a debugging session where we noticed that there were extents that had been written with the wrong checksum type. That's pretty bad - the extent we noticed was written with _no_ checksum, meaning IO path options got screwed up somewhere and we had no way of detecting it.

So - that means we need to be able to check and enforce that data is correctly written according to the specified IO path options from the filesystem and inode; something we've been needing for awhile.

IO path options -> extent handling is rebalance's responsibility (and one of these days I'm going to rename that, that name has become quite the inaccurate description for what it does). We already have the ability to store IO path options in the extent itself (struct bch_extent_rebalance); this is necessary for e.g. indirect extents where we don't know what inode it came from when doing background processing, as well as for the triggers so that we can do accounting for pending rebalance work.

So a large part of the series is reworking how we apply IO path options to extents (bch2_bkey_set_needs_rebalance()), so that we can strictly enforce that the extent does match the IO path options and record an error if it does not.

Additionally, in the current code rebalance only really handles the compression/background_compression/background_target options; it now needs to handle (enforce, apply, correct existing data) all the IO path options - that means we need to add support for the data_checksum, data_replicas and erasure_code option.

Supporting data_replicas requires reworking bch_extent_rebalance (compatibility notes in that patch), since the triggers that update the rebalance_work accounting and btrees (now btrees) need to be pure functions of the extents - they can't be functions of device state or durability, since those can change.

So bch_extent_rebalance now has flags for "extent needs to be rebalanced"; the trigger only looks at those (and this will benefit extent btree update performance); this centralizes looking at the entire extent and deciding what needs to be done in bch2_bkey_set_needs_rebalance(), which now has the new consistency checks. We now clearly define under what conditions and codepaths an extent is allowed to deviate: option changes, or a foreground write that only needs background_compression or background_target applied.

The fun stuff:

Now that rebalance can react to any IO path option changes,

- It's no longer required to run 'bcachefs data rereplicate/bcachefs data op drop_extra_replicas' after replicas setting changes, BCH_DATA_OP_rereplicate and BCH_DATA_OP_drop_extra_replicas are obsoleted.

- We can now react to device state changes: durability setting changes, and more importantly, state changes when a device switches to BCH_MEMBER_STATE_failed - this will automatically evacuate the device (and evacuate will resume if we crash or shutdown and restart).

Currently, we only mark devices read-only on excessive IO errors, we don't automatically mark devices as failed - that only happens in response to a 'bcachefs device set-state' command. But in the future we'll want to add configuration and policy for making this happen automatically when a device appears to be unhealthy.

If you have a huge disk array, this will mean no wearing a pager to respond to hardware failures: we'll do everything required to keep data at the appropriate replication level and on healthy devices, just swap the bad devices at your leisure.

Other good stuff:

- rebalance_work accounting is now broken out into subcounters for each IO path option so you can better see what background processing is happening

- new btrees for rebalance_work_hipri, to ensure that evacuating failed devices runs first, and rebalance_work_pending for data we'd like to move but can't because the target is full - this will solve the "rebalance is spinning because I tried to stuff more data into background_target than fits" bug reports.

Kent Overstreet (21):
bcachefs: s/bch_io_opts/bch_inode_opts
bcachefs: Inode opt helper refactoring
bcachefs: opt_change_cookie
bcachefs: Transactional consistency for set_needs_rebalance
bcachefs: Plumb bch_inode_opts.change_cookie
bcachefs: enum set_needs_rebalance_ctx
bcachefs: do_rebalance_scan() now responsible for indirect extents
bcachefs: Rename, split out bch2_extent_get_io_opts()
bcachefs: do_rebalance_extent() uses bch2_extent_get_apply_io_opts()
bcachefs: Correct propagation of io options to indirect extents
bcachefs: bkey_should_have_rb_opts()
bcachefs: bch2_bkey_needs_rebalance()
bcachefs: rebalance now supports changing checksum type
bcachefs: Consistency checking for bch_extent_rebalance opts
bcachefs: check_rebalance_work checks option inconsistency
bcachefs: bch2_bkey_set_needs_rebalance() now takes
per_snapshot_io_opts
bcachefs: bch_extent_rebalance changes
bcachefs: bch2_set_rebalance_needs_scan_device()
bcachefs: next_rebalance_extent() now handles replicas changes
bcachefs: rebalance: erasure_code opt change now handled
bcachefs: rebalance_work_(hipri|pending) btrees

 fs/bcachefs/bcachefs.h               |   2 +
 fs/bcachefs/bcachefs_format.h        |   8 +
 fs/bcachefs/buckets.c                |  28 +-
 fs/bcachefs/checksum.h               |   2 +-
 fs/bcachefs/data_update.c            |  44 +-
 fs/bcachefs/data_update.h            |   8 +-
 fs/bcachefs/disk_accounting_format.h |   1 +
 fs/bcachefs/extents.c                |  16 +-
 fs/bcachefs/extents.h                |  15 +-
 fs/bcachefs/fs-io-buffered.c         |  12 +-
 fs/bcachefs/fs-io-direct.c           |   8 +-
 fs/bcachefs/fs-io.c                  |   4 +-
 fs/bcachefs/inode.c                  |  45 +-
 fs/bcachefs/inode.h                  |   9 +-
 fs/bcachefs/io_misc.c                |  14 +-
 fs/bcachefs/io_misc.h                |   2 +-
 fs/bcachefs/io_read.c                |   2 +-
 fs/bcachefs/io_read.h                |   4 +-
 fs/bcachefs/io_write.c               |  50 +-
 fs/bcachefs/io_write.h               |   4 +-
 fs/bcachefs/io_write_types.h         |   2 +-
 fs/bcachefs/move.c                   | 194 +------
 fs/bcachefs/move.h                   |  34 +-
 fs/bcachefs/opts.c                   |  15 +-
 fs/bcachefs/opts.h                   |   9 +-
 fs/bcachefs/rebalance.c              | 823 +++++++++++++++++++++------
 fs/bcachefs/rebalance.h              |  80 ++-
 fs/bcachefs/rebalance_format.h       |  62 +-
 fs/bcachefs/reflink.c                |  16 +-
 fs/bcachefs/sb-errors_format.h       |   3 +-
 fs/bcachefs/super.c                  |   4 +
 fs/bcachefs/sysfs.c                  |   4 +
 fs/bcachefs/trace.h                  |   5 -
 fs/bcachefs/xattr.c                  |   3 +
 34 files changed, 945 insertions(+), 587 deletions(-)
Signature · 1 line
2.50.1