data chunk_size ceiling and shrinking stripe sizes on large RAID1/10 arrays
Hi all,
Because the default
/sys/fs/btrfs/<fsid>/allocation/data/chunk_size
value is capped at 10 GiB (and cannot be raised), the allocator’s logic in decide_stripe_size_regular() progressively reduces the per-device stripe_size once data_stripes * stripe_size would exceed that 10 GiB ceiling.
Concrete example:
40-device RAID10 → data_stripes = 20, allocator shrinks stripe_size to ≈ 512 MiB
Functionally, I think everything works, but the result is a much smaller stripe per device than the customary 1 GiB. With even more drives the stripe continues to decrease.
Questions
1. Is this behaviour considered acceptable or just a stop-gap until a better solution is developed? The limit seems.to have been introduced in https://github.com/torvalds/linux/commit/fce466eab7ac6baa9d2dcd88abcf945be3d4a089 (btrfs: tree-checker: Verify block_group_item) but with no direct explanation to why 10GiB was chosen.
2. What are the risks if we increase BTRFS_MAX_DATA_CHUNK_SIZE limit beyond 10GiB?
3. Has anyone measured performance impacts once stripe_size drops below 1 GiB? Are there other issues that may happen on such a filesystem?
Thanks!
~ Forza
Quoted 11 lines · Forza
Hi all,
Because the default
/sys/fs/btrfs/<fsid>/allocation/data/chunk_size
value is capped at 10 GiB (and cannot be raised), the allocator’s logic in
decide_stripe_size_regular()progressively reduces the per-devicestripe_sizeoncedata_stripes*stripe_sizewould exceed that 10 GiB ceiling.Concrete example:
40-device RAID10 →
data_stripes= 20, allocator shrinksstripe_sizeto ≈ 512 MiB
40 devices, this is already something we do not expect.
Maybe limiting the device number of a RAID0/RAID5/RAID6 chunk to 10/11/12 will be better?
Quoted 6 lines
Functionally, I think everything works, but the result is a much smaller stripe per device than the customary 1 GiB. With even more drives the stripe continues to decrease.
Questions
1. Is this behaviour considered acceptable or just a stop-gap until a better solution is developed? The limit seems.to have been introduced in https://github.com/torvalds/linux/commit/fce466eab7ac6baa9d2dcd88abcf945be3d4a089 (btrfs: tree-checker: Verify
block_group_item) but with no direct explanation to why 10GiB was chosen.
The 10GiB is a long existing limits, mostly from the beginning of btrfs, thus that tree check patch just keeps it, and that's also a good indicator of obviously wrong chunk size.
Quoted 2 lines
2. What are the risks if we increase
BTRFS_MAX_DATA_CHUNK_SIZElimit beyond 10GiB?
Further slowing down balance I believe.
Especially the balance itself is single threaded read, won't fully utilize all that many devices in RAID10.
Quoted 2 lines
3. Has anyone measured performance impacts once
stripe_sizedrops below 1 GiB? Are there other issues that may happen on such a filesystem?
No real world benchmark yet, but I guess it should be no big deal, as the stripe size is also limited by 10% of the device size.
Thus for fs < 10 GiB, their stripe size is already smaller than 1GiB.
Thanks,
Qu
Quoted 6 lines
Thanks!
~ Forza