Skip to content

btrfs: defrag: rework to support sector perfect defrag

Qu Wenruo linux-btrfs oe-kbuild oe-kbuild-all 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest11 patches 8 repliesIn review
  2. v410 patches 6 repliesSuperseded
  3. v310 patches 6 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v210 patches2 repliesSuperseded
v17 patches6 repliesSuperseded

Changes in v5 from the cover letter

- Fix the btrfs/072 test failure
Now btrfs/072 no longer reports inode nbytes mismatch error

- Add one patch to make cluster_pages_for_defrag() to be subpage compatible

- Move the page writeback wait out of the page preparation loop To keep the same old behavior.

- Use pgoff_t for page index

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11btrfs: defrag: pass file_ra_state instead of file for btrfs_defrag_file()fs/btrfs/ctree.h fs/btrfs/ioctl.c+20 -110 replies
2/11btrfs: defrag: also check PagePrivate for subpage cases in cluster_pages_for_defrag()fs/btrfs/ioctl.c+3 -20 replies
3/11btrfs: defrag: replace hard coded PAGE_SIZE to sectorsizefs/btrfs/ioctl.c+7 -50 replies
4/11btrfs: defrag: extract the page preparation code into one helperfs/btrfs/ioctl.c+88 -611 reply
5/11btrfs: defrag: introduce a new helper to collect target file extentsfs/btrfs/ioctl.c+120 -01 reply
6/11btrfs: defrag: introduce a helper to defrag a continuous prepared rangefs/btrfs/ioctl.c+56 -00 replies
7/11btrfs: defrag: introduce a helper to defrag a rangefs/btrfs/ioctl.c+95 -101 reply
8/11btrfs: defrag: introduce a new helper to defrag one clusterfs/btrfs/ioctl.c+56 -01 reply
9/11btrfs: defrag: use defrag_one_cluster() to implement btrfs_defrag_file()fs/btrfs/ioctl.c+55 -1492 replies
10/11btrfs: defrag: remove the old infrastructurefs/btrfs/ioctl.c+0 -3130 replies
11/11btrfs: defrag: enable defrag for subpage casefs/btrfs/ioctl.c+0 -60 replies

Reviews trailers per patch

1 of 11 patches has a Reviewed-by or Acked-by.

PatchJohannes Thumshirn
1/11R-b
2/11
3/11
4/11
5/11
6/11
7/11
8/11
9/11
10/11
11/11

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 00/11] btrfs: defrag: rework to support sector perfect defrag

This branch is based on David's misc-next branch, which already has the subpage read-write support included.

[BACKGROUND]
In current misc-next branch, we disable defrag completely due to the fact that current code can only work on page basis.

Without proper subpage defrag support, it could lead to problems like btrfs/062 crash.

Thus this patchset will make defrag to work on both regular and subpage sectorsize.

[SOLUTION]
To defrag a file range, what we do is pretty much like buffered write, except we don't really write any new data to page cache, but just mark the range dirty.

Then let later writeback to merge the range into a larger extent.

But current defrag code is working on per-page basis, not per-sector, thus we have to refactor it a little to make it to work properly for subpage.

This patch will separate the code into 3 layers:
Layer 0: btrfs_defrag_file()
The defrag entrance.
Just do proper inode lock and split the file into page aligned 256K clusters to defrag.

Layer 1: defrag_one_cluster()
Will collect the initial targets file extents, and pass each continuous target to defrag_one_range().

Layer 2: defrag_one_range()
Will prepare the needed page and extent locking. Then re-check the range for real target list, as initial target list is not consistent as it doesn't hage page/extent locking to prevent hole punching.

Layer 3: defrag_one_locked_target()
The real work, to make the extent range defrag and update involved page status.

[BEHAVIOR CHANGE]
In the refactor, there is one behavior change:

- Defraged sector counter is based on the initial target list This is mostly to avoid the parameters to be passed too deep into defrag_one_locked_target().
Considering the accounting is not that important, we can afford some difference.

[PATCH STRUCTURE]
Patch 01~04: Small independent refactor to improve readability
Patch 05~09: Implement the more readable and subpage friendly defrag
Patch 10: Cleanup of old infrastructure
Patch 11: Enable defrag for subpage case

Now both regular sectorsize and subpage sectorsize can pass defrag test group.

[CHANGELOG]
v2:
- Make sure we won't defrag hole
This is done by re-collect the target list after have page and extent locked. So that we can have a consistent view of the extent map.

- Add a new layer to avoid variable naming bugs
Since we need to handle real target list inside defrag_one_range(), and that function has parameters like "start" and "len", while inside the loop we need things like "entry->start" and "entry->len", it has already caused hard to debug bugs during development.

Thus introduce a new layer, defrag_one_ragen() to prepare pages/extent lock then pass the entry to defrag_one_locked_target().

v3:
- Fix extent_state leak
Now we pass the @cached_state to defrag_one_locked_target() other than allowing it to allocate a new one.

This can be reproduced by enabling "TEST_FS_MODULE_RELOAD" environment variable for fstests and run "-g defrag" group.

- Fix a random hang in test cases like btrfs/062
Since defrag_one_range() will lock the extent range first, then call defrag_collect_targets(), which will in turn call defrag_lookup_extent() and lock extent range again.

This will cause a dead lock, and this only happens when the extent range is smaller than the original locked range. Thus sometimes the test can pass, but sometimes it can hang.

Fix it by teaching defrag_collect_targets() and defrag_lookup_extent() to skip extent lock for certain call sites.
Thus this needs some loops for btrfs/062.
The hang possibility is around 1/2 ~ 1/3 when run in a loop.

v4:
- Fix a callsite which can cause hang due to extent locking The call site is defrag_lookup_extent() in defrag_collect_tagets(), which has one call site called with extent lock hold. Thus it also need to pass the @locked parameter

- Fix a typo
"defraged" -> "defragged"

v5:
- Fix the btrfs/072 test failure
Now btrfs/072 no longer reports inode nbytes mismatch error

- Add one patch to make cluster_pages_for_defrag() to be subpage compatible

- Move the page writeback wait out of the page preparation loop To keep the same old behavior.

- Use pgoff_t for page index

Qu Wenruo (11):
btrfs: defrag: pass file_ra_state instead of file for
btrfs_defrag_file()
btrfs: defrag: also check PagePrivate for subpage cases in
cluster_pages_for_defrag()
btrfs: defrag: replace hard coded PAGE_SIZE to sectorsize
btrfs: defrag: extract the page preparation code into one helper
btrfs: defrag: introduce a new helper to collect target file extents
btrfs: defrag: introduce a helper to defrag a continuous prepared
range
btrfs: defrag: introduce a helper to defrag a range
btrfs: defrag: introduce a new helper to defrag one cluster
btrfs: defrag: use defrag_one_cluster() to implement
btrfs_defrag_file()
btrfs: defrag: remove the old infrastructure
btrfs: defrag: enable defrag for subpage case

 fs/btrfs/ctree.h |   4 +-
 fs/btrfs/ioctl.c | 911 ++++++++++++++++++++++-------------------------
 2 files changed, 429 insertions(+), 486 deletions(-)
Signature · 1 line
2.32.0