Skip to content

btrfs: dynamic and periodic block_group reclaim

Boris Burkov linux-btrfs 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest6 patches 6 repliesIn review
  2. v16 patches 3 repliesSuperseded

Changes in v2 from the cover letter

- add reclaim errors counter
- refactor reclaim counter to remove extra else
- account for zone unusable in threshold calculation

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6btrfs: report reclaim stats in sysfsfs/btrfs/block-group.c fs/btrfs/space-info.h fs/btrfs/sysfs.c+34 -00 replies
2/6btrfs: store fs_info on space_infofs/btrfs/space-info.c fs/btrfs/space-info.h+2 -00 replies
3/6btrfs: dynamic block_group reclaim thresholdfs/btrfs/block-group.c fs/btrfs/space-info.c fs/btrfs/space-info.h +1 more+164 -201 reply
4/6btrfs: periodic block_group reclaimfs/btrfs/block-group.c fs/btrfs/block-group.h fs/btrfs/space-info.c +2 more+95 -00 replies
5/6btrfs: prevent pathological periodic reclaim loopsfs/btrfs/block-group.c fs/btrfs/space-info.c fs/btrfs/space-info.h+71 -114 replies
6/6btrfs: urgent periodic reclaim passfs/btrfs/space-info.c+34 -10 replies

Reviews trailers per patch

2 of 6 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchJohannes ThumshirnJosef Bacik
SeriesR-b
1/6R-b
2/6R-b
3/6
4/6
5/6
6/6

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/6] btrfs: dynamic and periodic block_group reclaim

Btrfs's block_group allocator suffers from a well known problem, that it is capable of eagerly allocating too much space to either data or metadata (most often data, absent bugs) and then later be unable to allocate more space for the other, when needed. When data starves metadata, this can extra painfully result in read only filesystems that need careful manual balancing to fix.

This can be worked around by:
- enabling automatic reclaim
- periodically running balance

The latter is widely deployed via btrfsmaintenance
(https://github.com/kdave/btrfsmaintenance) and the former is used at scale at Meta with good results. However, neither of those solutions is perfect, as they both currently use a fixed threshold. A fixed threshold is vulnerable to workloads that trigger high amounts of reclaim. This has led to btrfsmaintenance setting very conservative thresholds of 5 and 10 percent of data block groups.
(https://github.com/kdave/btrfsmaintenance/commit/edbbfffe592f47c2849a8825f523e2ccc38b15f5)
At Meta, we deal with an elevated level of reclaim which would be desirable to reduce.

This patch set expands on automatic reclaim, adding the ability to set a dynamic reclaim threshold that appropriately scales with the global file system allocation conditions as well as periodic reclaim which runs that reclaim sweep in the cleaner thread. Together, I believe they constitute a robust and general automatic reclaim system that should avoid unfortunate read only filesystems in all but extreme conditions, where space is running quite low anyway and failure is more reasonable.

At a very high level, the dynamic threshold's strategy is to set a fixed target of unallocated block groups (10 block groups) and linearly scale its aggression the further we are from that target. That way we do no automatic reclaim until we actually press against the unallocated target, allowing the allocator to gradually fill fragmented space with new extents, but do claw back space after workloads that use and free a bunch of space, perhaps with fragmentation.

I ran it on three workloads (described in detail on the dynamic reclaim patch) but they are:
1. bounce allocations around X% full.
2. fill up all the way and introduce full fragmentation.
3. write in a fragmented way until the filesystem is just about full. script can be found here:
https://github.com/boryas/scripts/tree/main/fio/reclaim

The important results can be seen here (full results explorable at
https://bur.io/dyn-rec/)

bounce at 30%, higher relocations with a fixed threshold:
https://bur.io/dyn-rec/bounce/reclaims.png
https://bur.io/dyn-rec/bounce/reclaim_bytes.png
https://bur.io/dyn-rec/bounce/unalloc_bytes.png

hard 30% fragmentation, dynamic actually reclaims, relocs not crazy:
https://bur.io/dyn-rec/strict_frag/reclaims.png
https://bur.io/dyn-rec/strict_frag/reclaim_bytes.png
https://bur.io/dyn-rec/strict_frag/unalloc_bytes.png

fill it all the way up in a fragmented way, then keep making allocations:
https://bur.io/dyn-rec/last_gig/reclaims.png
https://bur.io/dyn-rec/last_gig/reclaim_bytes.png
https://bur.io/dyn-rec/last_gig/unalloc_bytes.png
--
Changelog:
v2:
- add reclaim errors counter
- refactor reclaim counter to remove extra else
- account for zone unusable in threshold calculation

Boris Burkov (6):
btrfs: report reclaim stats in sysfs
btrfs: store fs_info on space_info
btrfs: dynamic block_group reclaim threshold
btrfs: periodic block_group reclaim
btrfs: prevent pathological periodic reclaim loops
btrfs: urgent periodic reclaim pass

 fs/btrfs/block-group.c |  42 ++++++--
 fs/btrfs/block-group.h |   1 +
 fs/btrfs/space-info.c  | 240 +++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/space-info.h  |  48 +++++++++
 fs/btrfs/sysfs.c       |  83 +++++++++++++-
 5 files changed, 391 insertions(+), 23 deletions(-)
Signature · 1 line
2.45.2