Skip to contentSkip to conversation

Btrfs updates for 5.9

GIT PULL Merged David Sterba 2 participants 2 messages Jul 31, 2020 – Aug 3, 2020 Btrfs file system

lore
Share & tools
lore

David Sterba

author maintainer
#0 lore raw

Hi,

we don't have any big feature updates this time, there are lots of small enhacements or fixes. A hilight perhaps are the parallel fsync performance improvements, numbers below.

Regarding the dio/iomap that was reverted last time, the required API changes are likely to land in the upcoming cycle, the btrfs part will be updated afterwards.

Please pull, thanks.

---

User visible changes:

- new mount option rescue= to group all recovery-related mount options so we don't have many specific options, currently introducing only aliases for existing options, future extensions are in development to allow read-only mount with partially damaged structures
- usebackuproot is an alias for rescue=usebackuproot
- nologreplay is an alias for rescue=nologreplay

- start deprecation of mount option inode_cache, removal scheduled to 5.11

- removed deprecated mount options alloc_start and subvolrootid

- device stats corruption counter gets incremented when a checksum mismatch is found

- qgroup information exported in sysfs /sys/fs/btrfs/<UUID>/qgroups/<id>

- add link /sys/fs/btrfs/<UUID>/bdi pointing to the associated backing dev info

- FS_INFO ioctl enhancements:
- add flags to request/describe newly added items
- new item: numeric checksum type and checksum size
- new item: generation
- new item: metadata_uuid

- seed device: with one new read-write device added, print the new device information in /proc/mounts

- balance: detect cancellation by Ctrl-C in existing cancellation points

Performance improvements:

- optimized versions of various helpers on little-endian architectures, where we don't have to do LE/BE conversion from on-disk format

- tree-log/fsync optimizations leading to lower max latency reported by dbench, reduced by about 12%

- all chunk tree leaves are prefetched at mount time, can improve mount time on large (terabyte-sized) filesystems

- speed up parallel fsync of files with reflinked/deduped extents, with jobs 16 to 1024 the throughput gets improved roughly by 50% on average and runtime decreased roughly by 30% on average, notable outlier is 128 jobs with +121.2% on throughput and -54.6% runtime

- another speed up of parallel fsync, reduce number of checksum tree lookups and contention, the improvements start to show up with 2 tasks with +20% throughput and -16% runtime up to 64 with +200% throughput and -66% runtime

Core:

- umount-time qgroup leak checker

- qgroups
- add a way to unreserve partial range after failure, avoiding some EDQUOT errors
- improved flushing logic when EDQUOT is hit

- possible EINTR interruption caused by failed reservations after transaction start is better handled and documented

- transaction abort errors are unified to EROFS in case it's not the original reason of abort or we don't have other way to determine the reason

Fixes:

- make truncate succeed on a NOCOW file even if data space is exhausted

- fix cancelling balance on filesystem with exhausted metadata space

- anon block device:
- preallocate anon bdev when subvolume is created to report failure early
- shorten time the anon bdev id is allocated
- don't allocate anon bdev for internal roots

- minor memory leak in ref-verify

- refuse invalid combinations of compression and NOCOW file flags

- lockdep fixes, updating the device locks

- remove obsolete fallback logic for block group profile adjustments when switching from 1 to more devices, causing allocation of unwanted block groups

Other:

- cleanups, refactoring, simplifications
- conversions from struct inode to struct btrfs_inode in internal functions
- removal of unused struct members

----------------------------------------------------------------
The following changes since commit 92ed301919932f777713b9172e525674157e983d:

Linux 5.8-rc7 (2020-07-26 14:14:06 -0700)

are available in the Git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-5.9-tag

for you to fetch changes up to 5e548b32018d96c377fda4bdac2bf511a448ca67:

btrfs: do not set the full sync flag on the inode during page release (2020-07-27 12:55:48 +0200)

----------------------------------------------------------------
Anand Jain (3):
btrfs: let btrfs_return_cluster_to_free_space() return void
btrfs: use helper btrfs_get_block_group
btrfs: don't traverse into the seed devices in show_devname

David Sterba (18):
btrfs: scrub: remove kmap/kunmap of pages
btrfs: scrub: unify naming of page address variables
btrfs: scrub: simplify superblock checksum calculation
btrfs: scrub: remove temporary csum array in scrub_checksum_super
btrfs: scrub: clean up temporary page variables in scrub_checksum_super
btrfs: scrub: simplify data block checksum calculation
btrfs: scrub: clean up temporary page variables in scrub_checksum_data
btrfs: scrub: simplify tree block checksum calculation
btrfs: scrub: clean up temporary page variables in scrub_checksum_tree_block
btrfs: add little-endian optimized key helpers
btrfs: don't use UAPI types for fiemap callback
btrfs: remove unused btrfs_root::defrag_trans_start
btrfs: start deprecation of mount option inode_cache
btrfs: allow use of global block reserve for balance item deletion
btrfs: remove deprecated mount option alloc_start
btrfs: remove deprecated mount option subvolrootid
btrfs: prefetch chunk tree leaves at mount
btrfs: add missing check for nocow and compression inode flags

Denis Efremov (1):
btrfs: tests: remove if duplicate in __check_free_space_extents()

Filipe Manana (13):
btrfs: remove no longer necessary chunk mutex locking cases
btrfs: remove the start argument from btrfs_free_reserved_data_space_noquota()
btrfs: use btrfs_alloc_data_chunk_ondemand() when allocating space for relocation
btrfs: remove no longer used log_list member of struct btrfs_ordered_extent
btrfs: remove no longer used trans_list member of struct btrfs_ordered_extent
btrfs: only commit the delayed inode when doing a full fsync
btrfs: only commit delayed items at fsync if we are logging a directory
btrfs: stop incremening log_batch for the log root tree when syncing log
btrfs: remove no longer needed use of log_writers for the log root tree
btrfs: reduce contention on log trees when logging checksums
btrfs: fix race between page release and a fast fsync
btrfs: release old extent maps during page release
btrfs: do not set the full sync flag on the inode during page release

Johannes Thumshirn (7):
btrfs: get mapping tree directly from fsinfo in find_first_block_group
btrfs: factor out reading of bg from find_frist_block_group
btrfs: use free_root_extent_buffer to free root
btrfs: pass checksum type via BTRFS_IOC_FS_INFO ioctl
btrfs: add filesystem generation to FS_INFO ioctl
btrfs: add metadata_uuid to FS_INFO ioctl
btrfs: open-code remount flag setting in btrfs_remount

Josef Bacik (10):
btrfs: convert block group refcount to refcount_t
btrfs: don't WARN if we abort a transaction with EROFS
btrfs: document special case error codes for fs errors
btrfs: return EROFS for BTRFS_FS_STATE_ERROR cases
btrfs: sysfs: use NOFS for device creation
btrfs: open device without device_list_mutex
btrfs: move the chunk_mutex in btrfs_read_chunk_tree
btrfs: fix lockdep splat from btrfs_dump_space_info
btrfs: don't adjust bg flags and use default allocation profiles
btrfs: if we're restriping, use the target restripe profile

Liao Pingfang (1):
btrfs: check-integrity: remove unnecessary failure messages during memory allocation

Marcos Paulo de Souza (1):
btrfs: add multi-statement protection to btrfs_set/clear_and_info macros

Nikolay Borisov (74):
btrfs: don't balance btree inode pages from buffered write path
btrfs: read stripe len directly in btrfs_rmap_block
btrfs: simplify checks when adding excluded ranges
btrfs: make __btrfs_add_ordered_extent take struct btrfs_inode
btrfs: make get_extent_allocation_hint take btrfs_inode
btrfs: make btrfs_lookup_ordered_extent take btrfs_inode
btrfs: make btrfs_reloc_clone_csums take btrfs_inode
btrfs: make create_io_em take btrfs_inode
btrfs: make extent_clear_unlock_delalloc take btrfs_inode
btrfs: make btrfs_csum_one_bio takae btrfs_inode
btrfs: make __btrfs_drop_extents take btrfs_inode
btrfs: remove hole check in prealloc_file_extent_cluster
btrfs: perform data management operations outside of inode lock
btrfs: use for loop in prealloc_file_extent_cluster
btrfs: tracepoints: fix btrfs_trigger_flush symbolic string for flags
btrfs: tracepoints: fix extent type symbolic name print
btrfs: tracepoints: move FLUSH_ACTIONS define
btrfs: tracepoints: fix qgroup reservation type printing
btrfs: tracepoints: switch extent_io_tree_owner to using EM macro
btrfs: tracepoints: convert flush states to using EM macros
btrfs: make qgroup_free_reserved_data take btrfs_inode
btrfs: make __btrfs_qgroup_release_data take btrfs_inode
btrfs: make btrfs_qgroup_free_data take btrfs_inode
btrfs: make cow_file_range_inline take btrfs_inode
btrfs: make btrfs_add_ordered_extent take btrfs_inode
btrfs: make cow_file_range take btrfs_inode
btrfs: make btrfs_add_ordered_extent_compress take btrfs_inode
btrfs: make btrfs_submit_compressed_write take btrfs_inode
btrfs: make submit_compressed_extents take btrfs_inode
btrfs: make btrfs_qgroup_release_data take btrfs_inode
btrfs: make insert_reserved_file_extent take btrfs_inode
btrfs: make fallback_to_cow take btrfs_inode
btrfs: make run_delalloc_nocow take btrfs_inode
btrfs: make cow_file_range_async take btrfs_inode
btrfs: make btrfs_dec_test_first_ordered_pending take btrfs_inode
btrfs: make __endio_write_update_ordered take btrfs_inode
btrfs: make btrfs_cleanup_ordered_extents take btrfs_inode
btrfs: make inode_can_compress take btrfs_inode
btrfs: make inode_need_compress take btrfs_inode
btrfs: make need_force_cow take btrfs_inode
btrfs: make btrfs_run_delalloc_range take btrfs_inode
btrfs: make btrfs_add_ordered_extent_dio take btrfs_inode
btrfs: make btrfs_create_dio_extent take btrfs_inode
btrfs: make btrfs_new_extent_direct take btrfs_inode
btrfs: make __extent_writepage_io take btrfs_inode
btrfs: make writepage_delalloc take btrfs_inode
btrfs: make btrfs_set_extent_delalloc take btrfs_inode
btrfs: make btrfs_dirty_pages take btrfs_inode
btrfs: make btrfs_qgroup_reserve_data take btrfs_inode
btrfs: make btrfs_free_reserved_data_space_noquota take btrfs_fs_info
btrfs: make btrfs_free_reserved_data_space take btrfs_inode
btrfs: make btrfs_delalloc_release_space take btrfs_inode
btrfs: make btrfs_check_data_free_space take btrfs_inode
btrfs: make btrfs_delalloc_reserve_space take btrfs_inode
btrfs: remove BTRFS_I calls in btrfs_writepage_fixup_worker
btrfs: make prealloc_file_extent_cluster take btrfs_inode
btrfs: make btrfs_set_inode_last_trans take btrfs_inode
btrfs: make btrfs_qgroup_check_reserved_leak take btrfs_inode
btrfs: make get_state_failrec return failrec directly
btrfs: streamline btrfs_get_io_failure_record logic
btrfs: record btrfs_device directly in btrfs_io_bio
btrfs: don't check for btrfs_device::bdev in btrfs_end_bio
btrfs: increment device corruption error in case of checksum error
btrfs: remove needless ASSERT check of orig_bio in end_compressed_bio_read
btrfs: increment corrupt device counter during compressed read
btrfs: sysfs: add bdi link to the fsid directory
btrfs: always initialize btrfs_bio::tgtdev_map/raid_map pointers
btrfs: raid56: remove redundant device check in rbio_add_io_page
btrfs: raid56: assign bio in while() when using bio_list_pop
btrfs: raid56: use in_range where applicable
btrfs: raid56: don't opencode swap() in __raid_recover_end_io
btrfs: remove fail label in check_compressed_csum
btrfs: raid56: remove out label in __raid56_parity_recover
btrfs: remove done label in writepage_delalloc

Qu Wenruo (22):
btrfs: introduce "rescue=" mount option
btrfs: inode: refactor the parameters of insert_reserved_file_extent()
btrfs: inode: move qgroup reserved space release to the callers of insert_reserved_file_extent()
btrfs: file: reserve qgroup space after the hole punch range is locked
btrfs: change timing for qgroup reserved space for ordered extents to fix reserved space leak
btrfs: qgroup: catch reserved space leaks at unmount time
btrfs: allow btrfs_truncate_block() to fallback to nocow for data space reservation
btrfs: add comments for btrfs_check_can_nocow() and can_nocow_extent()
btrfs: refactor btrfs_check_can_nocow() into two variants
btrfs: use __u16 for the return value of btrfs_qgroup_level()
btrfs: qgroup: export qgroups in sysfs
btrfs: don't allocate anonymous block device for user invisible roots
btrfs: free anon block device right after subvolume deletion
btrfs: preallocate anon block device at first phase of snapshot creation
btrfs: qgroup: allow to unreserve range without releasing other ranges
btrfs: qgroup: try to flush qgroup space when we get -EDQUOT
btrfs: qgroup: remove ASYNC_COMMIT mechanism in favor of reserve retry-after-EDQUOT
btrfs: qgroup: free per-trans reserved space when a subvolume gets dropped
btrfs: relocation: allow signal to cancel balance
btrfs: avoid possible signal interruption of btrfs_drop_snapshot() on relocation tree
btrfs: relocation: review the call sites which can be interrupted by signal
btrfs: add comments for btrfs_reserve_flush_enum

Tom Rix (1):
btrfs: ref-verify: fix memory leak in add_block_entry

 fs/btrfs/block-group.c                 | 211 +++++--------
 fs/btrfs/block-group.h                 |   3 +-
 fs/btrfs/btrfs_inode.h                 |  11 +
 fs/btrfs/check-integrity.c             |  27 +-
 fs/btrfs/compression.c                 |  30 +-
 fs/btrfs/compression.h                 |   4 +-
 fs/btrfs/ctree.c                       |  17 ++
 fs/btrfs/ctree.h                       | 127 ++++++--
 fs/btrfs/delalloc-space.c              |  36 +--
 fs/btrfs/delalloc-space.h              |  10 +-
 fs/btrfs/disk-io.c                     |  92 +++++-
 fs/btrfs/disk-io.h                     |   2 +
 fs/btrfs/extent-io-tree.h              |   5 +-
 fs/btrfs/extent-tree.c                 |  17 +-
 fs/btrfs/extent_io.c                   | 241 ++++++++-------
 fs/btrfs/extent_io.h                   |   4 +-
 fs/btrfs/file-item.c                   |   4 +-
 fs/btrfs/file.c                        | 143 +++++----
 fs/btrfs/free-space-cache.c            |  23 +-
 fs/btrfs/free-space-cache.h            |   2 +-
 fs/btrfs/inode-map.c                   |   3 +-
 fs/btrfs/inode.c                       | 530 ++++++++++++++++++---------------
 fs/btrfs/ioctl.c                       |  86 +++++-
 fs/btrfs/ordered-data.c                |  63 ++--
 fs/btrfs/ordered-data.h                |  19 +-
 fs/btrfs/qgroup.c                      | 359 ++++++++++++++++------
 fs/btrfs/qgroup.h                      |  24 +-
 fs/btrfs/raid56.c                      |  65 ++--
 fs/btrfs/ref-verify.c                  |   2 +
 fs/btrfs/reflink.c                     |  26 +-
 fs/btrfs/relocation.c                  |  71 ++---
 fs/btrfs/scrub.c                       | 153 +++-------
 fs/btrfs/space-info.c                  |   2 +-
 fs/btrfs/super.c                       | 144 ++++++---
 fs/btrfs/sysfs.c                       | 163 +++++++++-
 fs/btrfs/sysfs.h                       |   7 +
 fs/btrfs/tests/free-space-tree-tests.c |   2 -
 fs/btrfs/tests/inode-tests.c           |  14 +-
 fs/btrfs/transaction.c                 |   8 +-
 fs/btrfs/transaction.h                 |  28 +-
 fs/btrfs/tree-defrag.c                 |   5 +-
 fs/btrfs/tree-log.c                    |  50 ++--
 fs/btrfs/volumes.c                     | 133 ++++++---
 fs/btrfs/volumes.h                     |   2 +-
 include/trace/events/btrfs.h           | 137 +++++----
 include/uapi/linux/btrfs.h             |  21 +-
 include/uapi/linux/btrfs_tree.h        |   4 +-
 47 files changed, 1909 insertions(+), 1221 deletions(-)