Btrfs updates for 6.2
David Sterba
maintainerHi,
this round there are a lot of cleanups and moved code so the diffstat looks huge, otherwise there are some nice performance improvements and an update to raid56 reliability. Please pull, thanks.
---
User visible features:
- raid56 reliability vs performance trade off
- fix destructive RMW for raid5 data (raid6 still needs work) - do full RMW cycle for writes and verify all checksums before overwrite, this should prevent rewriting potentially corrupted data without notice
- stripes are cached in memory which should reduce the performance impact but still can hurt some workloads
- checksums are verified after repair again
- this is the last option without introducing additional features (write intent bitmap, journal, another tree), the RMW cycle was supposed to be avoided by the original implementation exactly for performance reasons but that caused all the reliability problems
- discard=async by default for devices that support it
- implement emergency flush reserve to avoid almost all unnecessary transaction aborts due to ENOSPC in cases where there are too many delayed refs or delayed allocation
- skip block group synchronization if there's no change in used bytes, can reduce transaction commit count for some workloads
Performance improvements:
- fiemap and lseek
- overall speedup due to skipping unnecessary or duplicate searches (-40% run time)
- cache some data structures and sharedness of extents (-30% run time)
- send
- faster backref resolution when finding clones
- cached leaf to root mapping for faster backref walking
- improved clone/sharing detection
- overall run time improvements (-70%)
Core:
- module initialization converted to a table of function pointers run in a sequence
- preparation for fscrypt, extend passing file names across calls, dir item can store encryption status
- raid56 updates
- more accurate error tracking of sectors within stripe
- simplify recovery path and remove dedicated endio worker kthread
- simplify scrub call paths
- refactoring to support the full RMW write
- tree block parentness checks consolidated and done at metadata read time
- improved error handling
- cleanups
- move a lot of code for better synchronization between kernel and user space sources, split big files
- enum cleanups
- GFP flag cleanups
- header file cleanups, prototypes, dependencies
- redundant parameter cleanups
- inline extent handling simplifications
- inode parameter conversion
- data structure cleanups, reductions, renames, merges
----------------------------------------------------------------
The following changes since commit 76dcd734eca23168cb008912c0f69ff408905235:
Linux 6.1-rc8 (2022-12-04 14:48:12 -0800)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.2-tag
for you to fetch changes up to b7af0635c87ff78d6bd523298ab7471f9ffd3ce5:
btrfs: print transaction aborted messages with an error level (2022-12-05 18:00:59 +0100)
----------------------------------------------------------------
Anand Jain (2):
btrfs: merge module cleanup sequence to one helper
btrfs: move device->name RCU allocation and assign to btrfs_alloc_device()
Artem Chernyshev (1):
btrfs: replace strncpy() with strscpy()
Boris Burkov (2):
btrfs: skip reclaim if block_group is empty
btrfs: re-check reclaim condition in reclaim worker
ChenXiaoSong (1):
btrfs: add might_sleep() annotations
Christoph Hellwig (3):
btrfs: move struct btrfs_tree_parent_check out of disk-io.h
btrfs: split the bio submission path into a separate file
btrfs: move repair_io_failure to bio.c
David Sterba (63):
btrfs: add helper for bit enumeration
btrfs: convert BTRFS_ILOCK-* defines to enum bit
btrfs: convert extent_io page op defines to enum bits
btrfs: convert EXTENT_* bits to enums
btrfs: convert QGROUP_* defines to enum bits
btrfs: convert __TRANS_* defines to enum bits
btrfs: sysfs: convert remaining scnprintf to sysfs_emit
btrfs: auto enable discard=async when possible
btrfs: simplify generation check in btrfs_get_dentry
btrfs: sink gfp_t parameter to btrfs_backref_iter_alloc
btrfs: sink gfp_t parameter to btrfs_qgroup_trace_extent
btrfs: switch GFP_NOFS to GFP_KERNEL in scrub_setup_recheck_block
btrfs: sink gfp_t parameter to alloc_scrub_sector
btrfs: update function comments
btrfs: fix SPDX comment in tree-mod-log.h
btrfs: simplify percent calculation helpers, rename div_factor
btrfs: switch extent_page_data bit fields to bools
btrfs: merge struct extent_page_data to btrfs_bio_ctrl
btrfs: zlib: use copy_page for full page copy
btrfs: zoned: use helper to check a power of two zone size
btrfs: convert btrfs_block_group::needs_free_space to runtime flag
btrfs: convert btrfs_block_group::seq_zone to runtime flag
btrfs: change how repair action is passed to btrfs_repair_one_sector
btrfs: drop parameter compression_type from btrfs_submit_dio_repair_bio
btrfs: change how submit bio callback is passed to btrfs_wq_submit_bio
btrfs: simplify btree_submit_bio_start and btrfs_submit_bio_start parameters
btrfs: switch async_submit_bio::inode to btrfs_inode
btrfs: pass btrfs_inode to btrfs_submit_bio_start
btrfs: pass btrfs_inode to btrfs_submit_bio_start_direct_io
btrfs: pass btrfs_inode to btrfs_wq_submit_bio
btrfs: pass btrfs_inode to btrfs_submit_metadata_bio
btrfs: pass btrfs_inode to btrfs_submit_data_write_bio
btrfs: pass btrfs_inode to btrfs_submit_data_read_bio
btrfs: pass btrfs_inode to btrfs_submit_dio_repair_bio
btrfs: pass btrfs_inode to submit_one_bio
btrfs: pass btrfs_inode to btrfs_repair_one_sector
btrfs: switch btrfs_dio_private::inode to btrfs_inode
btrfs: pass btrfs_inode to btrfs_submit_dio_bio
btrfs: pass btrfs_inode to btrfs_truncate
btrfs: pass btrfs_inode to btrfs_inode_lock
btrfs: pass btrfs_inode to btrfs_inode_unlock
btrfs: pass btrfs_inode to btrfs_dirty_inode
btrfs: pass btrfs_inode to btrfs_add_delalloc_inodes
btrfs: switch btrfs_writepage_fixup::inode to btrfs_inode
btrfs: pass btrfs_inode to btrfs_check_data_csum
btrfs: pass btrfs_inode to __unlink_start_trans
btrfs: pass btrfs_inode to btrfs_delete_subvolume
btrfs: drop private_data parameter from extent_io_tree_init
btrfs: switch extent_io_tree::private_data to btrfs_inode and rename
btrfs: pass btrfs_inode to btrfs_merge_delalloc_extent
btrfs: pass btrfs_inode to btrfs_set_delalloc_extent
btrfs: pass btrfs_inode to btrfs_split_delalloc_extent
btrfs: pass btrfs_inode to btrfs_clear_delalloc_extent
btrfs: pass btrfs_inode to btrfs_unlink_subvol
btrfs: pass btrfs_inode to btrfs_inode_by_name
btrfs: pass btrfs_inode to fixup_tree_root_location
btrfs: pass btrfs_inode to inode_tree_add
btrfs: pass btrfs_inode to btrfs_inherit_iflags
btrfs: switch async_chunk::inode to btrfs_inode
btrfs: use btrfs_inode inside compress_file_range
btrfs: use btrfs_inode inside btrfs_verify_data_csum
btrfs: pass btrfs_inode to btrfs_add_delayed_iput
btrfs: constify input buffer parameter in compression code
Filipe Manana (46):
btrfs: get the next extent map during fiemap/lseek more efficiently
btrfs: skip unnecessary extent map searches during fiemap and lseek
btrfs: skip unnecessary delalloc search during fiemap and lseek
btrfs: drop pointless memset when cloning extent buffer
btrfs: drop redundant bflags initialization when allocating extent buffer
btrfs: remove checks for a root with id 0 during backref walking
btrfs: remove checks for a 0 inode number during backref walking
btrfs: directly pass the inode to btrfs_is_data_extent_shared()
btrfs: turn the backref sharedness check cache into a context object
btrfs: move ulists to data extent sharedness check context
btrfs: remove roots ulist when checking data extent sharedness
btrfs: remove useless logic when finding parent nodes
btrfs: cache sharedness of the last few data extents during fiemap
btrfs: move up backref sharedness cache store and lookup functions
btrfs: avoid duplicated resolution of indirect backrefs during fiemap
btrfs: avoid unnecessary resolution of indirect backrefs during fiemap
btrfs: switch GFP_ATOMIC to GFP_NOFS when fixing up low keys
btrfs: remove gfp_t flag from btrfs_tree_mod_log_insert_key()
btrfs: update stale comment for nowait direct IO writes
btrfs: send: avoid unnecessary path allocations when finding extent clone
btrfs: send: update comment at find_extent_clone()
btrfs: send: drop unnecessary backref context field initializations
btrfs: send: avoid unnecessary backref lookups when finding clone source
btrfs: send: optimize clone detection to increase extent sharing
btrfs: use a single argument for extent offset in backref walking functions
btrfs: use a structure to pass arguments to backref walking functions
btrfs: reuse roots ulist on each leaf iteration for iterate_extent_inodes()
btrfs: constify ulist parameter of ulist_next()
btrfs: send: cache leaf to roots mapping during backref walking
btrfs: send: skip unnecessary backref iterations
btrfs: send: avoid double extent tree search when finding clone source
btrfs: send: skip resolution of our own backref when finding clone source
btrfs: send: bump the extent reference count limit for backref walking
btrfs: remove leftover setting of EXTENT_UPTODATE state in an inode's io_tree
btrfs: add an early exit when searching for delalloc range for lseek/fiemap
btrfs: skip unnecessary delalloc searches during lseek/fiemap
btrfs: search for delalloc more efficiently during lseek/fiemap
btrfs: remove no longer used btrfs_next_extent_map()
btrfs: allow passing a cached state record to count_range_bits()
btrfs: update stale comment for count_range_bits()
btrfs: use cached state when looking for delalloc ranges with fiemap
btrfs: use cached state when looking for delalloc ranges with lseek
btrfs: unify overwrite_item() and do_overwrite_item()
btrfs: remove outdated logic from overwrite_item() and add assertion
btrfs: do not BUG_ON() on ENOMEM when dropping extent items for a range
btrfs: print transaction aborted messages with an error level
Josef Bacik (91):
btrfs: add a cached_state to try_lock_extent
btrfs: use cached_state for btrfs_check_nocow_lock
btrfs: use a cached_state everywhere in relocation
btrfs: cache the failed state when locking extents
btrfs: add cached_state to read_extent_buffer_subpage
btrfs: remove unused set/clear_pending_info helpers
btrfs: remove unused BTRFS_TOTAL_BYTES_PINNED_BATCH
btrfs: remove unused BTRFS_IOPRIO_READA
btrfs: move btrfs on-disk definitions out of ctree.h
btrfs: move btrfs_get_block_group helper out of disk-io.h
btrfs: move maximum limits to btrfs_tree.h
btrfs: move BTRFS_MAX_MIRRORS into scrub.c
btrfs: move discard stat defs to free-space-cache.h
btrfs: move btrfs_should_fragment_free_space into block-group.c
btrfs: move flush related definitions to space-info.h
btrfs: move btrfs_print_data_csum_error into inode.c
btrfs: move trans_handle_cachep out of ctree.h
btrfs: move btrfs_path_cachep out of ctree.h
btrfs: move free space cachep's out of ctree.h
btrfs: move btrfs_next_old_item into ctree.c
btrfs: move the btrfs_verity_descriptor_item defs up in ctree.h
btrfs: introduce BTRFS_RESERVE_FLUSH_EMERGENCY
btrfs: do not use GFP_ATOMIC in the read endio
btrfs: remove unused unlock_extent_atomic
btrfs: do not panic if we can't allocate a prealloc extent state
btrfs: move fs wide helpers out of ctree.h
btrfs: move assert helpers out of ctree.h
btrfs: move the printk helpers out of ctree.h
btrfs: push printk index code into their respective helpers
btrfs: move BTRFS_FS_STATE* definitions and helpers to fs.h
btrfs: convert incompat and compat flag test helpers to macros
btrfs: move mount option definitions to fs.h
btrfs: move fs_info::flags enum to fs.h
btrfs: add a BTRFS_FS_NEED_TRANS_COMMIT flag
btrfs: remove fs_info::pending_changes and related code
btrfs: move the compat/incompat flag masks to fs.h
btrfs: rename struct-funcs.c to accessors.c
btrfs: move btrfs_map_token to accessors
btrfs: move accessor helpers into accessors.h
btrfs: remove temporary btrfs_map_token declaration in ctree.h
btrfs: move btrfs_fs_info declarations into fs.h
btrfs: move the lockdep helpers into locking.h
btrfs: minor whitespace in ctree.h
btrfs: remove extra space info prototypes in ctree.h
btrfs: move btrfs_account_ro_block_groups_free_space into space-info.c
btrfs: move extent-tree helpers into their own header file
btrfs: move delalloc space related prototypes to delalloc-space.h
btrfs: delete unused function prototypes in ctree.h
btrfs: move root tree prototypes to their own header
btrfs: remove unused function prototypes
btrfs: remove unused btrfs_cond_migrate_bytes
btrfs: convert discard stat defs to enum
btrfs: move btrfs_chunk_item_size out of ctree.h
btrfs: add dependencies to fs.h and block-rsv.h
btrfs: add blk_types.h include to compression.h
btrfs: move the printk and assert helpers to messages.c
btrfs: move inode prototypes to btrfs_inode.h
btrfs: rename tree-defrag.c to defrag.c
btrfs: move the auto defrag code to defrag.c
btrfs: move the file defrag code into defrag.c
btrfs: move defrag related prototypes to their own header
btrfs: move dir-item prototypes into dir-item.h
btrfs: move file-item prototypes into their own header
btrfs: move uuid tree prototypes to uuid-tree.h
btrfs: move ioctl prototypes into ioctl.h
btrfs: move file prototypes to file.h
btrfs: move the 32bit warn defines into messages.h
btrfs: move the snapshot drop related prototypes to extent-tree.h
btrfs: move acl prototypes into acl.h
btrfs: move relocation prototypes into relocation.h
btrfs: move scrub prototypes into scrub.h
btrfs: move dev-replace prototypes into dev-replace.h
btrfs: move verity prototypes into verity.h
btrfs: move CONFIG_BTRFS_FS_RUN_SANITY_TESTS checks to fs.h
btrfs: move super prototypes into super.h
btrfs: move super_block specific helpers into super.h
btrfs: move orphan prototypes into orphan.h
btrfs: move root helpers back into ctree.h
btrfs: move leaf_data_end into ctree.c
btrfs: move file_extent_item helpers into file-item.h
btrfs: move eb offset helpers into extent_io.h
btrfs: move the csum helpers into ctree.h
btrfs: pass the extent buffer for the btrfs_item_nr helpers
btrfs: add eb to btrfs_node_key_ptr_offset
btrfs: add helpers for manipulating leaf items and data
btrfs: remove BTRFS_LEAF_DATA_OFFSET
btrfs: add nr_global_roots to the super block definition
btrfs: add stack helpers for a few btrfs items
btrfs: fix uninitialized parent in insert_state
btrfs: fix uninitialized variable in find_first_clear_extent_bit
btrfs: sync some cleanups from progs into uapi/btrfs.h
Li zeming (1):
btrfs: allocate btrfs_io_context without GFP_NOFAIL
Omar Sandoval (1):
btrfs: extend btrfs_dir_item type to store encryption status
Peng Hao (1):
btrfs: simplify cleanup after error in btrfs_create_tree
Qu Wenruo (32):
btrfs: raid56: cleanup for function __free_raid_bio()
btrfs: raid56: allocate memory separately for rbio pointers
btrfs: raid56: make it more explicit that cache rbio should have all its data sectors uptodate
btrfs: make module init/exit match their sequence
btrfs: skip update of block group item if used bytes are the same
btrfs: selftests: remove impossible inline extent at non-zero file offset
btrfs: make inline extent read calculation much simpler
btrfs: do not reset extent map members for inline extents read
btrfs: remove new_inline argument from btrfs_extent_item_to_extent_map()
btrfs: extract the inline extent read code into its own function
btrfs: raid56: extract the vertical stripe recovery code into recover_vertical()
btrfs: raid56: extract the pq generation code into a helper
btrfs: raid56: extract the recovery bio list build code into a helper
btrfs: raid56: extract sector recovery code into a helper
btrfs: raid56: switch recovery path to a single function
btrfs: raid56: extract the rmw bio list build code into a helper
btrfs: raid56: extract rwm write bios assembly into a helper
btrfs: raid56: introduce the main entrance for RMW path
btrfs: raid56: switch write path to rmw_rbio()
btrfs: raid56: extract scrub read bio list assembly code into a helper
btrfs: raid56: switch scrub path to use a single function
btrfs: remove the unused endio_raid56_workers and btrfs_raid_bio::end_io_work
btrfs: raid56: introduce btrfs_raid_bio::error_bitmap
btrfs: raid56: migrate recovery and scrub recovery path to use error_bitmap
btrfs: raid56: remove the old error tracking system
btrfs: concentrate all tree block parentness check parameters into one structure
btrfs: move tree block parentness check into validate_extent_buffer()
btrfs: use btrfs_dev_name() helper to handle missing devices better
btrfs: refactor checksum calculations in btrfs_lookup_csums_range()
btrfs: introduce a bitmap based csum range search function
btrfs: raid56: prepare data checksums for later RMW verification
btrfs: raid56: do data csum verification during RMW cycle
Sweet Tea Dorminy (3):
btrfs: use struct qstr instead of name and namelen pairs
btrfs: setup qstr from dentrys using fscrypt helper
btrfs: use struct fscrypt_str instead of struct qstr
Wang Yugui (1):
btrfs: send add define for v2 buffer size
void0red (1):
btrfs: fix extent map use-after-free when handling missing device in read_one_chunk
fs/btrfs/Makefile | 6 +-
fs/btrfs/{struct-funcs.c => accessors.c} | 12 +-
fs/btrfs/accessors.h | 1073 ++++++++
fs/btrfs/acl.c | 2 +-
fs/btrfs/acl.h | 27 +
fs/btrfs/backref.c | 1001 +++++---
fs/btrfs/backref.h | 195 +-
fs/btrfs/bio.c | 381 +++
fs/btrfs/bio.h | 127 +
fs/btrfs/block-group.c | 152 +-
fs/btrfs/block-group.h | 30 +-
fs/btrfs/block-rsv.c | 43 +-
fs/btrfs/block-rsv.h | 6 +-
fs/btrfs/btrfs_inode.h | 161 +-
fs/btrfs/check-integrity.c | 4 +-
fs/btrfs/compression.c | 18 +-
fs/btrfs/compression.h | 11 +-
fs/btrfs/ctree.c | 311 ++-
fs/btrfs/ctree.h | 3927 ++----------------------------
fs/btrfs/defrag.c | 1376 +++++++++++
fs/btrfs/defrag.h | 22 +
fs/btrfs/delalloc-space.c | 61 +-
fs/btrfs/delalloc-space.h | 3 +
fs/btrfs/delayed-inode.c | 17 +-
fs/btrfs/delayed-inode.h | 2 +-
fs/btrfs/delayed-ref.c | 21 +-
fs/btrfs/dev-replace.c | 28 +-
fs/btrfs/dev-replace.h | 8 +
fs/btrfs/dir-item.c | 60 +-
fs/btrfs/dir-item.h | 42 +
fs/btrfs/discard.c | 112 +-
fs/btrfs/disk-io.c | 247 +-
fs/btrfs/disk-io.h | 35 +-
fs/btrfs/export.c | 25 +-
fs/btrfs/export.h | 3 +-
fs/btrfs/extent-io-tree.c | 192 +-
fs/btrfs/extent-io-tree.h | 100 +-
fs/btrfs/extent-tree.c | 55 +-
fs/btrfs/extent-tree.h | 78 +
fs/btrfs/extent_io.c | 482 ++--
fs/btrfs/extent_io.h | 67 +-
fs/btrfs/extent_map.c | 75 +-
fs/btrfs/file-item.c | 258 +-
fs/btrfs/file-item.h | 69 +
fs/btrfs/file.c | 621 ++---
fs/btrfs/file.h | 33 +
fs/btrfs/free-space-cache.c | 52 +-
fs/btrfs/free-space-cache.h | 13 +
fs/btrfs/free-space-tree.c | 15 +-
fs/btrfs/fs.c | 94 +
fs/btrfs/fs.h | 976 ++++++++
fs/btrfs/inode-item.c | 79 +-
fs/btrfs/inode-item.h | 20 +-
fs/btrfs/inode.c | 904 +++----
fs/btrfs/ioctl.c | 945 +------
fs/btrfs/ioctl.h | 17 +
fs/btrfs/locking.c | 1 +
fs/btrfs/locking.h | 76 +
fs/btrfs/lzo.c | 4 +-
fs/btrfs/messages.c | 353 +++
fs/btrfs/messages.h | 245 ++
fs/btrfs/misc.h | 24 +-
fs/btrfs/ordered-data.c | 25 +-
fs/btrfs/ordered-data.h | 3 +-
fs/btrfs/orphan.c | 1 +
fs/btrfs/orphan.h | 11 +
fs/btrfs/print-tree.c | 21 +-
fs/btrfs/props.c | 8 +-
fs/btrfs/props.h | 2 +-
fs/btrfs/qgroup.c | 78 +-
fs/btrfs/qgroup.h | 11 +-
fs/btrfs/raid56.c | 2066 ++++++++--------
fs/btrfs/raid56.h | 33 +-
fs/btrfs/rcu-string.h | 6 +-
fs/btrfs/ref-verify.c | 3 +
fs/btrfs/reflink.c | 30 +-
fs/btrfs/relocation.c | 94 +-
fs/btrfs/relocation.h | 23 +
fs/btrfs/root-tree.c | 24 +-
fs/btrfs/root-tree.h | 34 +
fs/btrfs/scrub.c | 75 +-
fs/btrfs/scrub.h | 16 +
fs/btrfs/send.c | 488 +++-
fs/btrfs/send.h | 6 +-
fs/btrfs/space-info.c | 86 +-
fs/btrfs/space-info.h | 78 +
fs/btrfs/subpage.c | 1 +
fs/btrfs/super.c | 554 +----
fs/btrfs/super.h | 29 +
fs/btrfs/sysfs.c | 16 +-
fs/btrfs/tests/btrfs-tests.c | 3 +-
fs/btrfs/tests/extent-buffer-tests.c | 1 +
fs/btrfs/tests/extent-io-tests.c | 4 +-
fs/btrfs/tests/free-space-tree-tests.c | 3 +-
fs/btrfs/tests/inode-tests.c | 58 +-
fs/btrfs/tests/qgroup-tests.c | 52 +-
fs/btrfs/transaction.c | 92 +-
fs/btrfs/transaction.h | 22 +-
fs/btrfs/tree-checker.c | 10 +-
fs/btrfs/tree-checker.h | 35 +-
fs/btrfs/tree-defrag.c | 132 -
fs/btrfs/tree-log.c | 452 ++--
fs/btrfs/tree-log.h | 5 +-
fs/btrfs/tree-mod-log.c | 36 +-
fs/btrfs/tree-mod-log.h | 4 +-
fs/btrfs/ulist.c | 38 +-
fs/btrfs/ulist.h | 2 +-
fs/btrfs/uuid-tree.c | 5 +-
fs/btrfs/uuid-tree.h | 12 +
fs/btrfs/verity.c | 6 +
fs/btrfs/verity.h | 28 +
fs/btrfs/volumes.c | 454 +---
fs/btrfs/volumes.h | 116 +-
fs/btrfs/xattr.c | 4 +
fs/btrfs/zlib.c | 6 +-
fs/btrfs/zoned.c | 18 +-
fs/btrfs/zoned.h | 1 +
fs/btrfs/zstd.c | 4 +-
include/trace/events/btrfs.h | 27 +-
include/uapi/linux/btrfs.h | 36 +-
include/uapi/linux/btrfs_tree.h | 235 ++
121 files changed, 11439 insertions(+), 9681 deletions(-)
rename fs/btrfs/{struct-funcs.c => accessors.c} (95%)
create mode 100644 fs/btrfs/accessors.h
create mode 100644 fs/btrfs/acl.h
create mode 100644 fs/btrfs/bio.c
create mode 100644 fs/btrfs/bio.h
create mode 100644 fs/btrfs/defrag.c
create mode 100644 fs/btrfs/defrag.h
create mode 100644 fs/btrfs/dir-item.h
create mode 100644 fs/btrfs/extent-tree.h
create mode 100644 fs/btrfs/file-item.h
create mode 100644 fs/btrfs/file.h
create mode 100644 fs/btrfs/fs.c
create mode 100644 fs/btrfs/fs.h
create mode 100644 fs/btrfs/ioctl.h
create mode 100644 fs/btrfs/messages.c
create mode 100644 fs/btrfs/messages.h
create mode 100644 fs/btrfs/orphan.h
create mode 100644 fs/btrfs/relocation.h
create mode 100644 fs/btrfs/root-tree.h
create mode 100644 fs/btrfs/scrub.h
create mode 100644 fs/btrfs/super.h
delete mode 100644 fs/btrfs/tree-defrag.c
create mode 100644 fs/btrfs/uuid-tree.h
create mode 100644 fs/btrfs/verity.hQuoted 6 lines · David Sterba
Hi,
this round there are a lot of cleanups and moved code so the diffstat looks huge, otherwise there are some nice performance improvements and an update to raid56 reliability. Please pull, thanks.
Quoted 8 lines
---
User visible features:
- raid56 reliability vs performance trade off
- fix destructive RMW for raid5 data (raid6 still needs work) - do full RMW cycle for writes and verify all checksums before overwrite, this should prevent rewriting potentially corrupted data without notice
Unfortunately, the "RMW" term seems abused.
I'd change the sentence into:
do full checksum verification for all data during RMW cycle.
Quoted 2 lines
- stripes are cached in memory which should reduce the performance impact but still can hurt some workloads
The cache behavior is not changed in this big chunk of raid56 work, but commit f6065f8edeb2 ("btrfs: raid56: don't trust any cached sector in __raid56_parity_recover()") is still the main thing affecting recovery path.
Thus although we didn't change the cache policy, it will still be bad for recovery cases (missing device, or some sector has mimsatch csum).
Quoted 5 lines
- checksums are verified after repair again
- this is the last option without introducing additional features (write intent bitmap, journal, another tree), the RMW cycle was supposed to be avoided by the original implementation exactly for performance reasons but that caused all the reliability problems
I don't know if the term "RMW" here is correctly used.
RMW is read-modify-write, which is the essential procedure for all RAID56 sub-stripe writes. Thus it's something can not be avoided, and the new code didn't avoid it at all.
The only new behavior is the extra verification during RMW.
(Thus it can be called RVMW, but I don't think anyone can recognize it anyway)
I'd change the "the RMW cycle ..." sentence into:
the extra checksum read/verification was supposed to be avoided by the original implementation exactly for performance reasons ...
Quoted 27 lines
- discard=async by default for devices that support it
- implement emergency flush reserve to avoid almost all unnecessary transaction aborts due to ENOSPC in cases where there are too many delayed refs or delayed allocation
- skip block group synchronization if there's no change in used bytes, can reduce transaction commit count for some workloads
Performance improvements:
- fiemap and lseek
- overall speedup due to skipping unnecessary or duplicate searches (-40% run time)
- cache some data structures and sharedness of extents (-30% run time)- send
- faster backref resolution when finding clones
- cached leaf to root mapping for faster backref walking
- improved clone/sharing detection
- overall run time improvements (-70%)Core:
- module initialization converted to a table of function pointers run in a sequence
Quoted 8 lines
- preparation for fscrypt, extend passing file names across calls, dir item can store encryption status
- raid56 updates
- more accurate error tracking of sectors within stripe
- simplify recovery path and remove dedicated endio worker kthread
- simplify scrub call paths
- refactoring to support the full RMW write
- refactoring to support the extra data checksum verification during RMW cycle.
Thanks,
Qu
Quoted 455 lines
- tree block parentness checks consolidated and done at metadata read time
- improved error handling
- cleanups
- move a lot of code for better synchronization between kernel and user space sources, split big files
- enum cleanups
- GFP flag cleanups
- header file cleanups, prototypes, dependencies
- redundant parameter cleanups
- inline extent handling simplifications
- inode parameter conversion
- data structure cleanups, reductions, renames, merges----------------------------------------------------------------
The following changes since commit76dcd734eca23168cb008912c0f69ff408905235:Linux 6.1-rc8 (2022-12-04 14:48:12 -0800)are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.gitfor-6.2-tagfor you to fetch changes up to
b7af0635c87ff78d6bd523298ab7471f9ffd3ce5:btrfs: print transaction aborted messages with an error level (2022-12-05 18:00:59 +0100)
----------------------------------------------------------------
Anand Jain (2):
btrfs: merge module cleanup sequence to one helper
btrfs: movedevice->nameRCU allocation and assign tobtrfs_alloc_device()Artem Chernyshev (1):
btrfs: replacestrncpy()withstrscpy()Boris Burkov (2):
btrfs: skip reclaim ifblock_groupis empty
btrfs: re-check reclaim condition in reclaim workerChenXiaoSong (1):
btrfs: addmight_sleep()annotationsChristoph Hellwig (3):
btrfs: move structbtrfs_tree_parent_checkout ofdisk-io.h
btrfs: split the bio submission path into a separate file
btrfs: moverepair_io_failuretobio.cDavid Sterba (63):
btrfs: add helper for bit enumeration
btrfs: convert BTRFS_ILOCK-* defines to enum bit
btrfs: convertextent_iopage op defines to enum bits
btrfs: convert EXTENT_* bits to enums
btrfs: convert QGROUP_* defines to enum bits
btrfs: convert __TRANS_* defines to enum bits
btrfs: sysfs: convert remaining scnprintf tosysfs_emit
btrfs: auto enable discard=async when possible
btrfs: simplify generation check inbtrfs_get_dentry
btrfs: sinkgfp_tparameter tobtrfs_backref_iter_alloc
btrfs: sinkgfp_tparameter tobtrfs_qgroup_trace_extent
btrfs: switchGFP_NOFStoGFP_KERNELinscrub_setup_recheck_block
btrfs: sinkgfp_tparameter toalloc_scrub_sector
btrfs: update function comments
btrfs: fix SPDX comment intree-mod-log.h
btrfs: simplify percent calculation helpers, renamediv_factor
btrfs: switchextent_page_databit fields to bools
btrfs: merge structextent_page_datatobtrfs_bio_ctrl
btrfs: zlib: usecopy_pagefor full page copy
btrfs: zoned: use helper to check a power of two zone size
btrfs: convertbtrfs_block_group::needs_free_spaceto runtime flag
btrfs: convertbtrfs_block_group::seq_zoneto runtime flag
btrfs: change how repair action is passed tobtrfs_repair_one_sector
btrfs: drop parametercompression_typefrombtrfs_submit_dio_repair_bio
btrfs: change how submit bio callback is passed tobtrfs_wq_submit_bio
btrfs: simplifybtree_submit_bio_startandbtrfs_submit_bio_startparameters
btrfs: switchasync_submit_bio::inodetobtrfs_inode
btrfs: passbtrfs_inodetobtrfs_submit_bio_start
btrfs: passbtrfs_inodetobtrfs_submit_bio_start_direct_io
btrfs: passbtrfs_inodetobtrfs_wq_submit_bio
btrfs: passbtrfs_inodetobtrfs_submit_metadata_bio
btrfs: passbtrfs_inodetobtrfs_submit_data_write_bio
btrfs: passbtrfs_inodetobtrfs_submit_data_read_bio
btrfs: passbtrfs_inodetobtrfs_submit_dio_repair_bio
btrfs: passbtrfs_inodetosubmit_one_bio
btrfs: passbtrfs_inodetobtrfs_repair_one_sector
btrfs: switchbtrfs_dio_private::inodetobtrfs_inode
btrfs: passbtrfs_inodetobtrfs_submit_dio_bio
btrfs: passbtrfs_inodetobtrfs_truncate
btrfs: passbtrfs_inodetobtrfs_inode_lock
btrfs: passbtrfs_inodetobtrfs_inode_unlock
btrfs: passbtrfs_inodetobtrfs_dirty_inode
btrfs: passbtrfs_inodetobtrfs_add_delalloc_inodes
btrfs: switchbtrfs_writepage_fixup::inodetobtrfs_inode
btrfs: passbtrfs_inodetobtrfs_check_data_csum
btrfs: passbtrfs_inodeto__unlink_start_trans
btrfs: passbtrfs_inodetobtrfs_delete_subvolume
btrfs: dropprivate_dataparameter fromextent_io_tree_init
btrfs: switchextent_io_tree::private_datatobtrfs_inodeand rename
btrfs: passbtrfs_inodetobtrfs_merge_delalloc_extent
btrfs: passbtrfs_inodetobtrfs_set_delalloc_extent
btrfs: passbtrfs_inodetobtrfs_split_delalloc_extent
btrfs: passbtrfs_inodetobtrfs_clear_delalloc_extent
btrfs: passbtrfs_inodetobtrfs_unlink_subvol
btrfs: passbtrfs_inodetobtrfs_inode_by_name
btrfs: passbtrfs_inodetofixup_tree_root_location
btrfs: passbtrfs_inodetoinode_tree_add
btrfs: passbtrfs_inodetobtrfs_inherit_iflags
btrfs: switchasync_chunk::inodetobtrfs_inode
btrfs: usebtrfs_inodeinsidecompress_file_range
btrfs: usebtrfs_inodeinsidebtrfs_verify_data_csum
btrfs: passbtrfs_inodetobtrfs_add_delayed_iput
btrfs: constify input buffer parameter in compression codeFilipe Manana (46):
btrfs: get the next extent map during fiemap/lseek more efficiently
btrfs: skip unnecessary extent map searches during fiemap and lseek
btrfs: skip unnecessary delalloc search during fiemap and lseek
btrfs: drop pointless memset when cloning extent buffer
btrfs: drop redundant bflags initialization when allocating extent buffer
btrfs: remove checks for a root with id 0 during backref walking
btrfs: remove checks for a 0 inode number during backref walking
btrfs: directly pass the inode tobtrfs_is_data_extent_shared()
btrfs: turn the backref sharedness check cache into a context object
btrfs: move ulists to data extent sharedness check context
btrfs: remove roots ulist when checking data extent sharedness
btrfs: remove useless logic when finding parent nodes
btrfs: cache sharedness of the last few data extents during fiemap
btrfs: move up backref sharedness cache store and lookup functions
btrfs: avoid duplicated resolution of indirect backrefs during fiemap
btrfs: avoid unnecessary resolution of indirect backrefs during fiemap
btrfs: switchGFP_ATOMICtoGFP_NOFSwhen fixing up low keys
btrfs: removegfp_tflag frombtrfs_tree_mod_log_insert_key()
btrfs: update stale comment for nowait direct IO writes
btrfs: send: avoid unnecessary path allocations when finding extent clone
btrfs: send: update comment atfind_extent_clone()
btrfs: send: drop unnecessary backref context field initializations
btrfs: send: avoid unnecessary backref lookups when finding clone source
btrfs: send: optimize clone detection to increase extent sharing
btrfs: use a single argument for extent offset in backref walking functions
btrfs: use a structure to pass arguments to backref walking functions
btrfs: reuse roots ulist on each leaf iteration foriterate_extent_inodes()
btrfs: constify ulist parameter ofulist_next()
btrfs: send: cache leaf to roots mapping during backref walking
btrfs: send: skip unnecessary backref iterations
btrfs: send: avoid double extent tree search when finding clone source
btrfs: send: skip resolution of our own backref when finding clone source
btrfs: send: bump the extent reference count limit for backref walking
btrfs: remove leftover setting ofEXTENT_UPTODATEstate in an inode'sio_tree
btrfs: add an early exit when searching for delalloc range for lseek/fiemap
btrfs: skip unnecessary delalloc searches during lseek/fiemap
btrfs: search for delalloc more efficiently during lseek/fiemap
btrfs: remove no longer usedbtrfs_next_extent_map()
btrfs: allow passing a cached state record tocount_range_bits()
btrfs: update stale comment forcount_range_bits()
btrfs: use cached state when looking for delalloc ranges with fiemap
btrfs: use cached state when looking for delalloc ranges with lseek
btrfs: unifyoverwrite_item()anddo_overwrite_item()
btrfs: remove outdated logic fromoverwrite_item()and add assertion
btrfs: do notBUG_ON()on ENOMEM when dropping extent items for a range
btrfs: print transaction aborted messages with an error levelJosef Bacik (91):
btrfs: add acached_statetotry_lock_extent
btrfs: usecached_stateforbtrfs_check_nocow_lock
btrfs: use acached_stateeverywhere in relocation
btrfs: cache the failed state when locking extents
btrfs: addcached_statetoread_extent_buffer_subpage
btrfs: remove unused set/clear_pending_info helpers
btrfs: remove unusedBTRFS_TOTAL_BYTES_PINNED_BATCH
btrfs: remove unusedBTRFS_IOPRIO_READA
btrfs: move btrfs on-disk definitions out ofctree.h
btrfs: movebtrfs_get_block_grouphelper out ofdisk-io.h
btrfs: move maximum limits tobtrfs_tree.h
btrfs: moveBTRFS_MAX_MIRRORSintoscrub.c
btrfs: move discard stat defs tofree-space-cache.h
btrfs: movebtrfs_should_fragment_free_spaceintoblock-group.c
btrfs: move flush related definitions tospace-info.h
btrfs: movebtrfs_print_data_csum_errorintoinode.c
btrfs: movetrans_handle_cachepout ofctree.h
btrfs: movebtrfs_path_cachepout ofctree.h
btrfs: move free space cachep's out ofctree.h
btrfs: movebtrfs_next_old_itemintoctree.c
btrfs: move thebtrfs_verity_descriptor_itemdefs up inctree.h
btrfs: introduceBTRFS_RESERVE_FLUSH_EMERGENCY
btrfs: do not useGFP_ATOMICin the read endio
btrfs: remove unusedunlock_extent_atomic
btrfs: do not panic if we can't allocate a prealloc extent state
btrfs: move fs wide helpers out ofctree.h
btrfs: move assert helpers out ofctree.h
btrfs: move the printk helpers out ofctree.h
btrfs: push printk index code into their respective helpers
btrfs: move BTRFS_FS_STATE* definitions and helpers tofs.h
btrfs: convert incompat and compat flag test helpers to macros
btrfs: move mount option definitions tofs.h
btrfs: movefs_info::flagsenum tofs.h
btrfs: add aBTRFS_FS_NEED_TRANS_COMMITflag
btrfs: removefs_info::pending_changesand related code
btrfs: move the compat/incompat flag masks tofs.h
btrfs: renamestruct-funcs.ctoaccessors.c
btrfs: movebtrfs_map_tokento accessors
btrfs: move accessor helpers intoaccessors.h
btrfs: remove temporarybtrfs_map_tokendeclaration inctree.h
btrfs: movebtrfs_fs_infodeclarations intofs.h
btrfs: move the lockdep helpers intolocking.h
btrfs: minor whitespace inctree.h
btrfs: remove extra space info prototypes inctree.h
btrfs: movebtrfs_account_ro_block_groups_free_spaceintospace-info.c
btrfs: move extent-tree helpers into their own header file
btrfs: move delalloc space related prototypes todelalloc-space.h
btrfs: delete unused function prototypes inctree.h
btrfs: move root tree prototypes to their own header
btrfs: remove unused function prototypes
btrfs: remove unusedbtrfs_cond_migrate_bytes
btrfs: convert discard stat defs to enum
btrfs: movebtrfs_chunk_item_sizeout ofctree.h
btrfs: add dependencies tofs.handblock-rsv.h
btrfs: addblk_types.hinclude tocompression.h
btrfs: move the printk and assert helpers tomessages.c
btrfs: move inode prototypes tobtrfs_inode.h
btrfs: renametree-defrag.ctodefrag.c
btrfs: move the auto defrag code todefrag.c
btrfs: move the file defrag code intodefrag.c
btrfs: move defrag related prototypes to their own header
btrfs: move dir-item prototypes intodir-item.h
btrfs: move file-item prototypes into their own header
btrfs: move uuid tree prototypes touuid-tree.h
btrfs: move ioctl prototypes intoioctl.h
btrfs: move file prototypes tofile.h
btrfs: move the 32bit warn defines intomessages.h
btrfs: move the snapshot drop related prototypes toextent-tree.h
btrfs: move acl prototypes intoacl.h
btrfs: move relocation prototypes intorelocation.h
btrfs: move scrub prototypes intoscrub.h
btrfs: move dev-replace prototypes intodev-replace.h
btrfs: move verity prototypes intoverity.h
btrfs: moveCONFIG_BTRFS_FS_RUN_SANITY_TESTSchecks tofs.h
btrfs: move super prototypes intosuper.h
btrfs: movesuper_blockspecific helpers intosuper.h
btrfs: move orphan prototypes intoorphan.h
btrfs: move root helpers back intoctree.h
btrfs: moveleaf_data_endintoctree.c
btrfs: movefile_extent_itemhelpers intofile-item.h
btrfs: move eb offset helpers intoextent_io.h
btrfs: move the csum helpers intoctree.h
btrfs: pass the extent buffer for thebtrfs_item_nrhelpers
btrfs: add eb tobtrfs_node_key_ptr_offset
btrfs: add helpers for manipulating leaf items and data
btrfs: removeBTRFS_LEAF_DATA_OFFSET
btrfs: addnr_global_rootsto the super block definition
btrfs: add stack helpers for a few btrfs items
btrfs: fix uninitialized parent ininsert_state
btrfs: fix uninitialized variable infind_first_clear_extent_bit
btrfs: sync some cleanups from progs intouapi/btrfs.hLi zeming (1):
btrfs: allocatebtrfs_io_contextwithoutGFP_NOFAILOmar Sandoval (1):
btrfs: extendbtrfs_dir_itemtype to store encryption statusPeng Hao (1):
btrfs: simplify cleanup after error inbtrfs_create_treeQu Wenruo (32):
btrfs: raid56: cleanup for function__free_raid_bio()
btrfs: raid56: allocate memory separately for rbio pointers
btrfs: raid56: make it more explicit that cache rbio should have all its data sectors uptodate
btrfs: make module init/exit match their sequence
btrfs: skip update of block group item if used bytes are the same
btrfs: selftests: remove impossible inline extent at non-zero file offset
btrfs: make inline extent read calculation much simpler
btrfs: do not reset extent map members for inline extents read
btrfs: removenew_inlineargument frombtrfs_extent_item_to_extent_map()
btrfs: extract the inline extent read code into its own function
btrfs: raid56: extract the vertical stripe recovery code intorecover_vertical()
btrfs: raid56: extract the pq generation code into a helper
btrfs: raid56: extract the recovery bio list build code into a helper
btrfs: raid56: extract sector recovery code into a helper
btrfs: raid56: switch recovery path to a single function
btrfs: raid56: extract the rmw bio list build code into a helper
btrfs: raid56: extract rwm write bios assembly into a helper
btrfs: raid56: introduce the main entrance for RMW path
btrfs: raid56: switch write path tormw_rbio()
btrfs: raid56: extract scrub read bio list assembly code into a helper
btrfs: raid56: switch scrub path to use a single function
btrfs: remove the unusedendio_raid56_workersandbtrfs_raid_bio::end_io_work
btrfs: raid56: introducebtrfs_raid_bio::error_bitmap
btrfs: raid56: migrate recovery and scrub recovery path to useerror_bitmap
btrfs: raid56: remove the old error tracking system
btrfs: concentrate all tree block parentness check parameters into one structure
btrfs: move tree block parentness check intovalidate_extent_buffer()
btrfs: usebtrfs_dev_name()helper to handle missing devices better
btrfs: refactor checksum calculations inbtrfs_lookup_csums_range()
btrfs: introduce a bitmap based csum range search function
btrfs: raid56: prepare data checksums for later RMW verification
btrfs: raid56: do data csum verification during RMW cycleSweet Tea Dorminy (3):
btrfs: use struct qstr instead of name and namelen pairs
btrfs: setup qstr from dentrys using fscrypt helper
btrfs: use structfscrypt_strinstead of struct qstrWang Yugui (1):
btrfs: send add define for v2 buffer sizevoid0red (1):
btrfs: fix extent map use-after-free when handling missing device inread_one_chunkfs/btrfs/Makefile | 6 +- fs/btrfs/{struct-funcs.c => accessors.c} | 12 +- fs/btrfs/accessors.h | 1073 ++++++++ fs/btrfs/acl.c | 2 +- fs/btrfs/acl.h | 27 + fs/btrfs/backref.c | 1001 +++++--- fs/btrfs/backref.h | 195 +- fs/btrfs/bio.c | 381 +++ fs/btrfs/bio.h | 127 + fs/btrfs/block-group.c | 152 +- fs/btrfs/block-group.h | 30 +- fs/btrfs/block-rsv.c | 43 +- fs/btrfs/block-rsv.h | 6 +- fs/btrfs/btrfs_inode.h | 161 +- fs/btrfs/check-integrity.c | 4 +- fs/btrfs/compression.c | 18 +- fs/btrfs/compression.h | 11 +- fs/btrfs/ctree.c | 311 ++- fs/btrfs/ctree.h | 3927 ++---------------------------- fs/btrfs/defrag.c | 1376 +++++++++++ fs/btrfs/defrag.h | 22 + fs/btrfs/delalloc-space.c | 61 +- fs/btrfs/delalloc-space.h | 3 + fs/btrfs/delayed-inode.c | 17 +- fs/btrfs/delayed-inode.h | 2 +- fs/btrfs/delayed-ref.c | 21 +- fs/btrfs/dev-replace.c | 28 +- fs/btrfs/dev-replace.h | 8 + fs/btrfs/dir-item.c | 60 +- fs/btrfs/dir-item.h | 42 + fs/btrfs/discard.c | 112 +- fs/btrfs/disk-io.c | 247 +- fs/btrfs/disk-io.h | 35 +- fs/btrfs/export.c | 25 +- fs/btrfs/export.h | 3 +- fs/btrfs/extent-io-tree.c | 192 +- fs/btrfs/extent-io-tree.h | 100 +- fs/btrfs/extent-tree.c | 55 +- fs/btrfs/extent-tree.h | 78 + fs/btrfs/extent_io.c | 482 ++-- fs/btrfs/extent_io.h | 67 +- fs/btrfs/extent_map.c | 75 +- fs/btrfs/file-item.c | 258 +- fs/btrfs/file-item.h | 69 + fs/btrfs/file.c | 621 ++--- fs/btrfs/file.h | 33 + fs/btrfs/free-space-cache.c | 52 +- fs/btrfs/free-space-cache.h | 13 + fs/btrfs/free-space-tree.c | 15 +- fs/btrfs/fs.c | 94 + fs/btrfs/fs.h | 976 ++++++++ fs/btrfs/inode-item.c | 79 +- fs/btrfs/inode-item.h | 20 +- fs/btrfs/inode.c | 904 +++---- fs/btrfs/ioctl.c | 945 +------ fs/btrfs/ioctl.h | 17 + fs/btrfs/locking.c | 1 + fs/btrfs/locking.h | 76 + fs/btrfs/lzo.c | 4 +- fs/btrfs/messages.c | 353 +++ fs/btrfs/messages.h | 245 ++ fs/btrfs/misc.h | 24 +- fs/btrfs/ordered-data.c | 25 +- fs/btrfs/ordered-data.h | 3 +- fs/btrfs/orphan.c | 1 + fs/btrfs/orphan.h | 11 + fs/btrfs/print-tree.c | 21 +- fs/btrfs/props.c | 8 +- fs/btrfs/props.h | 2 +- fs/btrfs/qgroup.c | 78 +- fs/btrfs/qgroup.h | 11 +- fs/btrfs/raid56.c | 2066 ++++++++-------- fs/btrfs/raid56.h | 33 +- fs/btrfs/rcu-string.h | 6 +- fs/btrfs/ref-verify.c | 3 + fs/btrfs/reflink.c | 30 +- fs/btrfs/relocation.c | 94 +- fs/btrfs/relocation.h | 23 + fs/btrfs/root-tree.c | 24 +- fs/btrfs/root-tree.h | 34 + fs/btrfs/scrub.c | 75 +- fs/btrfs/scrub.h | 16 + fs/btrfs/send.c | 488 +++- fs/btrfs/send.h | 6 +- fs/btrfs/space-info.c | 86 +- fs/btrfs/space-info.h | 78 + fs/btrfs/subpage.c | 1 + fs/btrfs/super.c | 554 +---- fs/btrfs/super.h | 29 + fs/btrfs/sysfs.c | 16 +- fs/btrfs/tests/btrfs-tests.c | 3 +- fs/btrfs/tests/extent-buffer-tests.c | 1 + fs/btrfs/tests/extent-io-tests.c | 4 +- fs/btrfs/tests/free-space-tree-tests.c | 3 +- fs/btrfs/tests/inode-tests.c | 58 +- fs/btrfs/tests/qgroup-tests.c | 52 +- fs/btrfs/transaction.c | 92 +- fs/btrfs/transaction.h | 22 +- fs/btrfs/tree-checker.c | 10 +- fs/btrfs/tree-checker.h | 35 +- fs/btrfs/tree-defrag.c | 132 - fs/btrfs/tree-log.c | 452 ++-- fs/btrfs/tree-log.h | 5 +- fs/btrfs/tree-mod-log.c | 36 +- fs/btrfs/tree-mod-log.h | 4 +- fs/btrfs/ulist.c | 38 +- fs/btrfs/ulist.h | 2 +- fs/btrfs/uuid-tree.c | 5 +- fs/btrfs/uuid-tree.h | 12 + fs/btrfs/verity.c | 6 + fs/btrfs/verity.h | 28 + fs/btrfs/volumes.c | 454 +--- fs/btrfs/volumes.h | 116 +- fs/btrfs/xattr.c | 4 + fs/btrfs/zlib.c | 6 +- fs/btrfs/zoned.c | 18 +- fs/btrfs/zoned.h | 1 + fs/btrfs/zstd.c | 4 +- include/trace/events/btrfs.h | 27 +- include/uapi/linux/btrfs.h | 36 +- include/uapi/linux/btrfs_tree.h | 235 ++ 121 files changed, 11439 insertions(+), 9681 deletions(-)rename
fs/btrfs/{struct-funcs.c=>accessors.c} (95%) create mode 100644fs/btrfs/accessors.hcreate mode 100644fs/btrfs/acl.h
create mode 100644fs/btrfs/bio.c
create mode 100644fs/btrfs/bio.h
create mode 100644fs/btrfs/defrag.c
create mode 100644fs/btrfs/defrag.h
create mode 100644fs/btrfs/dir-item.hcreate mode 100644fs/btrfs/extent-tree.hcreate mode 100644fs/btrfs/file-item.hcreate mode 100644fs/btrfs/file.h
create mode 100644fs/btrfs/fs.c
create mode 100644fs/btrfs/fs.h
create mode 100644fs/btrfs/ioctl.h
create mode 100644fs/btrfs/messages.ccreate mode 100644fs/btrfs/messages.hcreate mode 100644fs/btrfs/orphan.h
create mode 100644fs/btrfs/relocation.hcreate mode 100644fs/btrfs/root-tree.hcreate mode 100644fs/btrfs/scrub.h
create mode 100644fs/btrfs/super.h
delete mode 100644fs/btrfs/tree-defrag.ccreate mode 100644fs/btrfs/uuid-tree.hcreate mode 100644fs/btrfs/verity.h
Re: [GIT PULL] Btrfs updates for 6.2 (updated merge log)
Quoted 6 lines · Qu Wenruo
Quoted 4 lines
- raid56 reliability vs performance trade off
- fix destructive RMW for raid5 data (raid6 still needs work) - do full RMW cycle for writes and verify all checksums before overwrite, this should prevent rewriting potentially corrupted data without noticeUnfortunately, the "RMW" term seems abused.
I used is as a shortcut but it's probably confusing, thanks for the suggested updates.
Quoted 9 lines
Quoted 2 lines
- stripes are cached in memory which should reduce the performance impact but still can hurt some workloads
The cache behavior is not changed in this big chunk of raid56 work, but commit
f6065f8edeb2("btrfs: raid56: don't trust any cached sector in__raid56_parity_recover()") is still the main thing affecting recovery path.Thus although we didn't change the cache policy, it will still be bad for recovery cases (missing device, or some sector has mimsatch csum).
Yeah, there's no change but I felt it should be mentioned together with the RMW as it'll be used more than before.
Linus, below is the complete merge log with the edits.
---
User visible features:
- raid56 reliability vs performance trade off
- fix destructive RMW for raid5 data (raid6 still needs work) - do full checksum verification for all data during RMW cycle, this should prevent rewriting potentially corrupted data without notice
- stripes are cached in memory which should reduce the performance impact but still can hurt some workloads
- checksums are verified after repair again
- this is the last option without introducing additional features (write intent bitmap, journal, another tree), the extra checksum read/verification was supposed to be avoided by the original implementation exactly for performance reasons but that caused all the reliability problems
- discard=async by default for devices that support it
- implement emergency flush reserve to avoid almost all unnecessary transaction aborts due to ENOSPC in cases where there are too many delayed refs or delayed allocation
- skip block group synchronization if there's no change in used bytes, can reduce transaction commit count for some workloads
Performance improvements:
- fiemap and lseek
- overall speedup due to skipping unnecessary or duplicate searches (-40% run time)
- cache some data structures and sharedness of extents (-30% run time)
- send
- faster backref resolution when finding clones
- cached leaf to root mapping for faster backref walking
- improved clone/sharing detection
- overall run time improvements (-70%)
Core:
- module initialization converted to a table of function pointers run in a sequence
- preparation for fscrypt, extend passing file names across calls, dir item can store encryption status
- raid56 updates
- more accurate error tracking of sectors within stripe
- simplify recovery path and remove dedicated endio worker kthread
- simplify scrub call paths
- refactoring to support the extra data checksum verification during RMW cycle
- tree block parentness checks consolidated and done at metadata read time
- improved error handling
- cleanups
- move a lot of code for better synchronization between kernel and user space sources, split big files
- enum cleanups
- GFP flag cleanups
- header file cleanups, prototypes, dependencies
- redundant parameter cleanups
- inline extent handling simplifications
- inode parameter conversion
- data structure cleanups, reductions, renames, merges
The pull request you sent on Mon, 12 Dec 2022 21:16:12 +0100:
The pull request you sent on Mon, 12 Dec 2022 21:16:12 +0100:
Quoted 1 line
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.gitfor-6.2-tag
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/149c51f876322d9bfbd5e2d6ffae7aff3d794384
Thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html