Skip to content

btrfs: add subpage support for RAID56

Qu Wenruo linux-btrfs 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest17 patches 16 repliesIn review
  2. v116 patches 16 repliesSuperseded

Changes in v2 from the cover letter

- Rebased to latest misc-next
There are several conflicts caused by bio interface change and page allocation update.

- A new patch to reduce the width of @stripe_len to u32 Currently @stripe_len is fixed to 64K, and even in the future we choose to enlarge the value, I see no reason to go beyond 4G for stripe length.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/17btrfs: reduce width for stripe_len from u64 to u32fs/btrfs/raid56.c fs/btrfs/raid56.h fs/btrfs/volumes.c +1 more+22 -270 replies
2/17btrfs: open-code rbio_nr_pages()fs/btrfs/raid56.c+5 -130 replies
3/17btrfs: make btrfs_raid_bio more compactfs/btrfs/raid56.c+22 -200 replies
4/17btrfs: introduce new cached members for btrfs_raid_biofs/btrfs/raid56.c+18 -53 replies
5/17btrfs: introduce btrfs_raid_bio::stripe_sectorsfs/btrfs/raid56.c+60 -50 replies
6/17btrfs: introduce btrfs_raid_bio::bio_sectorsfs/btrfs/raid56.c+49 -20 replies
7/17btrfs: make rbio_add_io_page() subpage compatiblefs/btrfs/raid56.c+161 -8611 replies
8/17btrfs: make finish_parity_scrub() subpage compatiblefs/btrfs/raid56.c+32 -240 replies
9/17btrfs: make __raid_recover_endio_io() subpage compatibablefs/btrfs/raid56.c+29 -230 replies
10/17btrfs: make finish_rmw() subpage compatiblefs/btrfs/raid56.c+32 -760 replies
11/17btrfs: open-code rbio_stripe_page_index()fs/btrfs/raid56.c+1 -70 replies
12/17btrfs: make raid56_add_scrub_pages() subpage compatiblefs/btrfs/raid56.c fs/btrfs/raid56.h fs/btrfs/scrub.c+12 -60 replies
13/17btrfs: remove btrfs_raid_bio::bio_pages arrayfs/btrfs/raid56.c+3 -350 replies
14/17btrfs: make set_bio_pages_uptodate() subpage compatiblefs/btrfs/raid56.c+39 -60 replies
15/17btrfs: make steal_rbio() subpage compatiblefs/btrfs/raid56.c+24 -30 replies
16/17btrfs: make alloc_rbio_essential_pages() subpage compatiblefs/btrfs/raid56.c+9 -70 replies
17/17btrfs: enable subpage support for RAID56fs/btrfs/disk-io.c fs/btrfs/raid56.c fs/btrfs/volumes.c+0 -210 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/17] btrfs: add subpage support for RAID56

The branch can be fetched from github, based on latest misc-next branch (with bio and memory allocation refactors):
https://github.com/adam900710/linux/tree/subpage_raid56

[CHANGELOG]
v2:
- Rebased to latest misc-next
There are several conflicts caused by bio interface change and page allocation update.

- A new patch to reduce the width of @stripe_len to u32 Currently @stripe_len is fixed to 64K, and even in the future we choose to enlarge the value, I see no reason to go beyond 4G for stripe length.

Thus change it u32 to avoid some u64-divided-by-u32 situations.

This will reduce memory usage for map_lookup (which has a lifespan as long as the mounted fs) and btrfs_io_geometry (which only has a very short lifespan, mostly bounded to bio).

Furthermore, add some extra alignment check and use right bit shift to replace involved division to avoid possible problems on 32bit systems.

- Pack sector_ptr::pgoff and sector_ptr::uptodate into one u32 This will reduce memory usage and reduce unaligned memory access

Please note that, even with it packed, we still have a 4 bytes padding (it's u64 + u32, thus not perfectly aligned).
Without packed attribute, it will cost more memory usage anyway.

- Call kunmap_local() using address with pgoff
As it can handle it without problem, no need to bother extra search just for pgoff.

- Use "= { 0 }" for structure initialization

- Reduce comment updates to minimal
If one comment line is not really touched, then don't touch it just to fix some bad styles.

[DESIGN]
To make RAID56 code to support subpage, we need to make every sector of a RAID56 full stripe (including P/Q) to be addressable.

Previously we use page pointer directly for things like stripe_pages:

Full stripe is 64K * 3, 2 data stripes, one P stripe (RAID5)

stripe_pages: | 0 | 1 | 2 | .. | 46 | 47 |

Those 48 pages all points to a page we allocated.

The new subpage support will introduce a sector layer, based on the old stripe_pages[] array:

The same 64K * 3, RAID5 layout, but 64K page size and 4K sector size:

stripe_sectors: | 0 | 1 | .. |15 |16 |  ...  |31 |32 | ..    |47 |
stripe_pages:   |      Page 0    |     Page 1    |    Page 2     |

Each stripe_ptr of stripe_sectors[] will include:

- One page pointer
Points back the page inside stripe_pages[].

- One pgoff member
To indicate the offset inside the page

- One uptodate member
To indicate if the sector is uptodate, replacing the old PageUptodate flag.
As introducing btrfs_subpage structure to stripe_pages[] looks a little overkilled, as we only care Uptodate flag.

The same applies to bio_sectors[] array, which is going to completely replace the old bio_pages[] array.

[SIDE EFFECT]
Despite the obvious new ability for subpage to utilize btrfs RAID56 profiles, it will cause more memory usage for real btrfs_raid_bio structure.

We allocate extra memory based on the stripe size and number of stripes, and update the pointer arrays to utilize the extra memory.

To compare, we use a pretty standard setup, 3 disks raid5, 4K page size on x86_64:

Before: 1176
After: 2320 (+97.8%)

The reason for such a big bump is:

- Extra size for sector_ptr.
Instead of just a page pointer, now it's twice the size of a pointer (a page pointer + 2 * unsigned int)

This means although we replace bio_pages[] with bio_sectors[], we are still enlarging the size.

- A completely new array for stripe_sectors[]
And the array itself is also twice the size of the old stripe_pages[].

- Extra padding for sector_ptr
Since we don't have packed attribute anymore, the real size of a sector_ptr is 16 bytes, not 12 bytes.

There is some attempt to reduce the size of btrfs_raid_bio itself, but the big impact still comes from the new sector_ptr arrays.

Without exotic macros or packed attribute, I don't have any better ideas on reducing the real size of btrfs_raid_bio.

[TESTS]
Now due to recent new error path exposed in generic/475, only btrfs groups are tested. Or it will always hang at generic/475 due to unrelated bugs.

Both x86_64 and aarch64 (64K page size) pass the full btrfs test cases without new regression.

[PATCHSET LAYOUT]
The patchset layout puts several things into consideration:

- Every patch can be tested independently on x86_64 No more tons of unused helpers then a big switch. Every change can be verified on x86_64.

- More temporary sanity checks than previous code
For example, when rbio_add_io_page() is converted to be subpage compatible, extra ASSERT() is added to ensure no subpage range can even be added.

Such temporary checks are removed in the last enablement patch. This is to make testing on x86_64 more comprehensive.

- Mostly small change in each patch
The only exception is the conversion for rbio_add_io_page(). But the most change in that patch comes from variable renaming. The overall line changed in each patch should still be small enough for review.

Qu Wenruo (17):
btrfs: reduce width for stripe_len from u64 to u32
btrfs: open-code rbio_nr_pages()
btrfs: make btrfs_raid_bio more compact
btrfs: introduce new cached members for btrfs_raid_bio
btrfs: introduce btrfs_raid_bio::stripe_sectors
btrfs: introduce btrfs_raid_bio::bio_sectors
btrfs: make rbio_add_io_page() subpage compatible
btrfs: make finish_parity_scrub() subpage compatible
btrfs: make __raid_recover_endio_io() subpage compatibable
btrfs: make finish_rmw() subpage compatible
btrfs: open-code rbio_stripe_page_index()
btrfs: make raid56_add_scrub_pages() subpage compatible
btrfs: remove btrfs_raid_bio::bio_pages array
btrfs: make set_bio_pages_uptodate() subpage compatible
btrfs: make steal_rbio() subpage compatible
btrfs: make alloc_rbio_essential_pages() subpage compatible
btrfs: enable subpage support for RAID56

 fs/btrfs/disk-io.c |   8 -
 fs/btrfs/raid56.c  | 749 +++++++++++++++++++++++++++------------------
 fs/btrfs/raid56.h  |   8 +-
 fs/btrfs/scrub.c   |   6 +-
 fs/btrfs/volumes.c |  27 +-
 fs/btrfs/volumes.h |   8 +-
 6 files changed, 479 insertions(+), 327 deletions(-)
Signature · 1 line
2.35.1