xfsprogs: catch us up to 6.10
Hi Carlos,
Please pull these fully reviewed patchsets for xfsprogs 6.10.
--D
[GIT PULL 01/23] libxfs: fixes for 6.9
Hi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 42fc61ebd061ffc63a0302a8422ed21f67e38b5f:
xfsprogs: Release v6.9.0 (2024-07-22 11:05:03 +0200)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/libxfs-6.9-fixes_2024-07-29
for you to fetch changes up to c6438c3e740e139adcd6c236606498d49bbb5655:
xfs_repair: don't crash on -vv (2024-07-29 17:00:59 -0700)
----------------------------------------------------------------
libxfs: fixes for 6.9 [01/28]
A couple more last minute fixes for 6.9.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Chris Hofstaedtler (1):
Remove support for split-/usr installs
Christoph Hellwig (1):
repair: btree blocks are never on the RT subvolume
Darrick J. Wong (3):
xfile: fix missing error unlock in xfile_fcb_findxfs_repair: don't leak the rootdir inode when orphanage already existsxfs_repair: don't crash on -vv
configure.ac | 21 --------------------- debian/Makefile | 4 ++-- debian/local/initramfs.hook | 2 +- debian/rules | 5 ++--- fsck/Makefile | 4 ++-- include/builddefs.in | 2 -- include/buildmacros | 20 ++++++++++---------- libxfs/xfile.c | 6 +++--- mkfs/Makefile | 4 ++-- repair/Makefile | 4 ++-- repair/phase6.c | 7 +++++-- repair/progress.c | 2 +- repair/scan.c | 21 +++++---------------- repair/xfs_repair.c | 2 +- 14 files changed, 36 insertions(+), 68 deletions(-)
[GIT PULL 02/23] libxfs: sync with 6.10
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit c6438c3e740e139adcd6c236606498d49bbb5655:
xfs_repair: don't crash on -vv (2024-07-29 17:00:59 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/libxfs-sync-6.10_2024-07-29
for you to fetch changes up to 7fbf8e036dc1d5b9caaf6f64ad4bc88d40c8292b:
xfs: fix direction in XFS_IOC_EXCHANGE_RANGE (2024-07-29 17:01:05 -0700)
----------------------------------------------------------------
libxfs: sync with 6.10 [02/28]
Synchronize libxfs with the kernel.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Allison Henderson (14):
xfs: Increase XFS_DEFER_OPS_NR_INODES to 5
xfs: add parent pointer support to attribute code
xfs: define parent pointer ondisk extended attribute format
xfs: add parent pointer validator functions
xfs: extend transaction reservations for parent attributes
xfs: parent pointer attribute creation
xfs: add parent attributes to link
xfs: add parent attributes to symlink
xfs: remove parent pointers in unlink
xfs: Add parent pointers to rename
xfs: don't return XFS_ATTR_PARENT attributes via listxattr
xfs: pass the attr value to put_listent when possible
xfs: don't remove the attr fork when parent pointers are enabled
xfs: add a incompat feature bit for parent pointers
Christoph Hellwig (26):
xfs: make XFS_TRANS_LOWMODE match the other XFS_TRANS_ definitions
xfs: refactor realtime inode locking
xfs: free RT extents after updating the bmap btree
xfs: move RT inode locking out of __xfs_bunmapi
xfs: split xfs_mod_freecounter
xfs: reinstate RT support in xfs_bmapi_reserve_delalloc
xfs: cleanup fdblock/frextent accounting in xfs_bmap_del_extent_delay
xfs: support RT inodes in xfs_mod_delalloc
xfs: rework splitting of indirect block reservations
xfs: stop the steal (of data blocks for RT indirect blocks)
xfs: check the flags earlier in xfs_attr_match
xfs: factor out a xfs_dir_lookup_args helper
xfs: factor out a xfs_dir_createname_args helper
xfs: factor out a xfs_dir_removename_args helper
xfs: factor out a xfs_dir_replace_args helper
xfs: refactor dir format helpers
xfs: fix error returns from xfs_bmapi_write
xfs: remove the unusued tmp_logflags variable in xfs_bmapi_allocate
xfs: lift a xfs_valid_startblock into xfs_bmapi_allocate
xfs: don't open code XFS_FILBLKS_MIN in xfs_bmapi_write
xfs: pass the actual offset and len to allocate to xfs_bmapi_allocate
xfs: remove the xfs_iext_peek_prev_extent call in xfs_bmapi_allocate
xfs: fix xfs_bmap_add_extent_delay_real for partial conversions
xfs: do not allocate the entire delalloc extent in xfs_bmapi_write
xfs: xfs_quota_unreserve_blkres can't fail
xfs: simplify iext overflow checking and upgrade
Darrick J. Wong (70):
xfs: pass xfs_buf lookup flags to xfs_*read_agi
xfs: constify xfs_bmap_is_written_extent
xfs: introduce new file range exchange ioctl
xfs: create a incompat flag for atomic file mapping exchanges
xfs: introduce a file mapping exchange log intent item
xfs: create deferred log items for file mapping exchanges
xfs: add error injection to test file mapping exchange recovery
xfs: condense extended attributes after a mapping exchange operation
xfs: condense directories after a mapping exchange operation
xfs: condense symbolic links after a mapping exchange operation
xfs: make file range exchange support realtime files
xfs: capture inode generation numbers in the ondisk exchmaps log item
xfs: enable logged file mapping exchange feature
xfs: add an explicit owner field to xfs_da_args
xfs: use the xfs_da_args owner field to set new dir/attr block owner
xfs: validate attr leaf buffer owners
xfs: validate attr remote value buffer owners
xfs: validate dabtree node buffer owners
xfs: validate directory leaf buffer owners
xfs: validate explicit directory data buffer owners
xfs: validate explicit directory block buffer owners
xfs: validate explicit directory free block owners
xfs: use atomic extent swapping to fix user file fork data
xfs: repair extended attributes
xfs: expose xfs_bmap_local_to_extents for online repair
xfs: pass the owner to xfs_symlink_write_target
xfs: check unused nlink fields in the ondisk inode
xfs: try to avoid allocating from sick inode clusters
xfs: pin inodes that would otherwise overflow link count
xfs: remove XFS_DA_OP_REMOVE
xfs: remove XFS_DA_OP_NOTIME
xfs: remove xfs_da_args.attr_flags
xfs: make attr removal an explicit operation
xfs: rearrange xfs_da_args a bit to use less space
xfs: attr fork iext must be loaded before calling xfs_attr_is_leaf
xfs: fix missing check for invalid attr flags
xfs: restructure xfs_attr_complete_op a bit
xfs: use helpers to extract xattr op from opflags
xfs: enforce one namespace per attribute
xfs: rearrange xfs_attr_match parameters
xfs: move xfs_attr_defer_add to xfs_attr_item.c
xfs: create a separate hashname function for extended attributes
xfs: allow xattr matching on name and value for parent pointers
xfs: create attr log item opcodes and formats for parent pointers
xfs: record inode generation in xattr update log intent items
xfs: create a hashname function for parent pointers
xfs: split out handle management helpers a bit
xfs: add parent pointer ioctls
xfs: fix unit conversion error in xfs_log_calc_max_attrsetm_res
xfs: drop compatibility minimum log size computations for reflink
xfs: enable parent pointers
xfs: check dirents have parent pointers
xfs: remove some boilerplate from xfs_attr_set
xfs: make the reserved block permission flag explicit in xfs_attr_set
xfs: add raw parent pointer apis to support repair
xfs: remove pointless unlocked assertion
xfs: split xfs_bmap_add_attrfork into two pieces
xfs: actually rebuild the parent pointer xattrs
xfs: teach online scrub to find directory tree structure problems
xfs: report directory tree corruption in the health information
xfs: introduce vectored scrub mode
xfs: use unsigned ints for non-negative quantities in xfs_attr_remote.c
xfs: turn XFS_ATTR3_RMT_BUF_SPACE into a function
xfs: create a helper to compute the blockcount of a max sized remote value
xfs: minor cleanups of xfs_attr3_rmt_blocks
xfs: fix xfs_init_attr_trans not handling explicit operation codes
xfs: allow symlinks with short remote targets
xfs: restrict when we try to align cow fork delalloc to cowextsz hints
xfs: allow unlinked symlinks and dirs with zero size
xfs: fix direction in XFS_IOC_EXCHANGE_RANGE
John Garry (1):
xfs: Stop using __maybe_unused in xfs_alloc.c
Ritesh Harjani (IBM) (1):
xfs: Add cond_resched to block unmap range and reflink remap path
Wengang Wang (1):
xfs: make sure sb_fdblocks is non-negative
Zhang Yi (2):
xfs: make the seq argument to xfs_bmapi_convert_delalloc() optional
xfs: make xfs_bmapi_convert_delalloc() to allocate the target offset
db/attr.c | 2 +- db/attrset.c | 11 +- db/iunlink.c | 4 +- db/metadump.c | 8 +- db/namei.c | 23 +- include/libxfs.h | 1 + include/xfs_inode.h | 16 + include/xfs_mount.h | 2 + include/xfs_trace.h | 15 + io/inject.c | 1 + libxfs/Makefile | 5 + libxfs/defer_item.c | 116 ++++ libxfs/defer_item.h | 13 + libxfs/init.c | 3 + libxfs/libxfs_api_defs.h | 6 +- libxfs/libxfs_priv.h | 75 ++- libxfs/util.c | 6 + libxfs/xfs_ag.c | 12 +- libxfs/xfs_ag_resv.c | 24 +- libxfs/xfs_ag_resv.h | 2 +- libxfs/xfs_alloc.c | 10 +- libxfs/xfs_attr.c | 272 ++++++---- libxfs/xfs_attr.h | 49 +- libxfs/xfs_attr_leaf.c | 154 ++++-- libxfs/xfs_attr_leaf.h | 4 +- libxfs/xfs_attr_remote.c | 102 ++-- libxfs/xfs_attr_remote.h | 8 +- libxfs/xfs_attr_sf.h | 1 + libxfs/xfs_bmap.c | 409 +++++++------- libxfs/xfs_bmap.h | 13 +- libxfs/xfs_da_btree.c | 189 ++++++- libxfs/xfs_da_btree.h | 34 +- libxfs/xfs_da_format.h | 37 +- libxfs/xfs_defer.c | 12 +- libxfs/xfs_defer.h | 10 +- libxfs/xfs_dir2.c | 283 +++++----- libxfs/xfs_dir2.h | 23 +- libxfs/xfs_dir2_block.c | 42 +- libxfs/xfs_dir2_data.c | 18 +- libxfs/xfs_dir2_leaf.c | 100 +++- libxfs/xfs_dir2_node.c | 44 +- libxfs/xfs_dir2_priv.h | 15 +- libxfs/xfs_errortag.h | 4 +- libxfs/xfs_exchmaps.c | 1232 +++++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_exchmaps.h | 124 +++++ libxfs/xfs_format.h | 34 +- libxfs/xfs_fs.h | 158 +++++- libxfs/xfs_health.h | 4 +- libxfs/xfs_ialloc.c | 56 +- libxfs/xfs_ialloc.h | 5 +- libxfs/xfs_ialloc_btree.c | 4 +- libxfs/xfs_inode_buf.c | 55 +- libxfs/xfs_inode_fork.c | 63 +-- libxfs/xfs_inode_fork.h | 6 +- libxfs/xfs_log_format.h | 89 +++- libxfs/xfs_log_rlimit.c | 46 ++ libxfs/xfs_ondisk.h | 6 + libxfs/xfs_parent.c | 376 +++++++++++++ libxfs/xfs_parent.h | 110 ++++ libxfs/xfs_rtbitmap.c | 57 ++ libxfs/xfs_rtbitmap.h | 17 + libxfs/xfs_sb.c | 15 +- libxfs/xfs_shared.h | 6 +- libxfs/xfs_symlink_remote.c | 54 +- libxfs/xfs_symlink_remote.h | 8 +- libxfs/xfs_trans_resv.c | 324 ++++++++++-- libxfs/xfs_trans_space.c | 121 +++++ libxfs/xfs_trans_space.h | 29 +- mkfs/proto.c | 3 +- repair/attr_repair.c | 7 +- repair/phase6.c | 35 +- 71 files changed, 4349 insertions(+), 873 deletions(-)
create mode 100644 libxfs/xfs_exchmaps.c
create mode 100644 libxfs/xfs_exchmaps.h
create mode 100644 libxfs/xfs_parent.c
create mode 100644 libxfs/xfs_parent.h
create mode 100644 libxfs/xfs_trans_space.c
[GIT PULL 03/23] xfsprogs: atomic file updates
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 7fbf8e036dc1d5b9caaf6f64ad4bc88d40c8292b:
xfs: fix direction in XFS_IOC_EXCHANGE_RANGE (2024-07-29 17:01:05 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/atomic-file-updates-6.10_2024-07-29
for you to fetch changes up to 39e346ba525c51dd2f405ed5d6368db712fac586:
mkfs: add a formatting option for exchange-range (2024-07-29 17:01:06 -0700)
----------------------------------------------------------------
xfsprogs: atomic file updates [v30.9 03/28]
This series creates a new XFS_IOC_EXCHANGE_RANGE ioctl to exchange ranges of bytes between two files atomically.
This new functionality enables data storage programs to stage and commit file updates such that reader programs will see either the old contents or the new contents in their entirety, with no chance of torn writes. A successful call completion guarantees that the new contents will be seen even if the system fails.
The ability to exchange file fork mappings between files in this manner is critical to supporting online filesystem repair, which is built upon the strategy of constructing a clean copy of a damaged structure and committing the new structure into the metadata file atomically. The ioctls exist to facilitate testing of the new functionality and to enable future application program designs.
User programs will be able to update files atomically by opening an O_TMPFILE, reflinking the source file to it, making whatever updates they want to make, and exchange the relevant ranges of the temp file with the original file. If the updates are aligned with the file block size, a new (since v2) flag provides for exchanging only the written areas. Note that application software must quiesce writes to the file while it stages an atomic update. This will be addressed by a subsequent series.
This mechanism solves the clunkiness of two existing atomic file update
mechanisms: for O_TRUNC + rewrite, this eliminates the brief period where other programs can see an empty file. For create tempfile + rename, the need to copy file attributes and extended attributes for each file update is eliminated.
However, this method introduces its own awkwardness -- any program initiating an exchange now needs to have a way to signal to other programs that the file contents have changed. For file access mediated via read and write, fanotify or inotify are probably sufficient. For mmaped files, that may not be fast enough.
The reference implementation in XFS creates a new log incompat feature and log intent items to track high level progress of swapping ranges of two files and finish interrupted work if the system goes down. Sample code can be found in the corresponding changes to xfs_io to exercise the use case mentioned above.
Note that this function is /not/ the O_DIRECT atomic untorn file writes concept that has also been floating around for years. It is also not the RWF_ATOMIC patchset that has been shared. This RFC is constructed entirely in software, which means that there are no limitations other than the general filesystem limits.
As a side note, the original motivation behind the kernel functionality is online repair of file-based metadata. The atomic file content exchange is implemented as an atomic exchange of file fork mappings, which means that we can implement online reconstruction of extended attributes and directories by building a new one in another inode and exchanging the contents.
Subsequent patchsets adapt the online filesystem repair code to use atomic file exchanges. This enables repair functions to construct a clean copy of a directory, xattr information, symbolic links, realtime bitmaps, and realtime summary information in a temporary inode. If this completes successfully, the new contents can be committed atomically into the inode being repaired. This is essential to avoid making corruption problems worse if the system goes down in the middle of running repair.
For userspace, this series also includes the userspace pieces needed to test the new functionality, and a sample implementation of atomic file updates.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (12):
man: document the exchange-range ioctl
man: document XFS_FSOP_GEOM_FLAGS_EXCHRANGE
libhandle: add support for bulkstat v5
libfrog: add support for exchange range ioctl familyxfs_db: advertise exchange-range in the version commandxfs_logprint: support dumping exchmaps log itemsxfs_fsr: convert to bulkstat v5 ioctlsxfs_fsr: skip the xattr/forkoff levering with the newer swapext implementationsxfs_io: create exchangerange command to test file range exchange ioctl
libfrog: advertise exchange-range supportxfs_repair: add exchange-range to file systems
mkfs: add a formatting option for exchange-range
db/sb.c | 2 + fsr/xfs_fsr.c | 167 +++++++++++++--------- include/jdm.h | 23 +++ io/Makefile | 48 ++++++- io/exchrange.c | 156 ++++++++++++++++++++ io/init.c | 1 + io/io.h | 1 + libfrog/Makefile | 2 + libfrog/file_exchange.c | 52 +++++++ libfrog/file_exchange.h | 15 ++ libfrog/fsgeom.c | 49 +++++-- libfrog/fsgeom.h | 1 + libhandle/jdm.c | 117 +++++++++++++++ logprint/log_misc.c | 11 ++ logprint/log_print_all.c | 12 ++ logprint/log_redo.c | 128 +++++++++++++++++ logprint/logprint.h | 6 + man/man2/ioctl_xfs_exchange_range.2 | 278 ++++++++++++++++++++++++++++++++++++ man/man2/ioctl_xfs_fsgeometry.2 | 3 + man/man8/mkfs.xfs.8.in | 7 + man/man8/xfs_admin.8 | 7 + man/man8/xfs_io.8 | 40 ++++++ mkfs/lts_4.19.conf | 1 + mkfs/lts_5.10.conf | 1 + mkfs/lts_5.15.conf | 1 + mkfs/lts_5.4.conf | 1 + mkfs/lts_6.1.conf | 1 + mkfs/lts_6.6.conf | 1 + mkfs/xfs_mkfs.c | 26 +++- repair/globals.c | 1 + repair/globals.h | 1 + repair/phase2.c | 30 ++++ repair/xfs_repair.c | 11 ++ 33 files changed, 1114 insertions(+), 87 deletions(-)
create mode 100644 io/exchrange.c
create mode 100644 libfrog/file_exchange.c create mode 100644 libfrog/file_exchange.h create mode 100644 man/man2/ioctl_xfs_exchange_range.2
[GIT PULL 04/23] xfsprogs: set and validate dir/attr block owners
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 39e346ba525c51dd2f405ed5d6368db712fac586:
mkfs: add a formatting option for exchange-range (2024-07-29 17:01:06 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/dirattr-validate-owners-6.10_2024-07-29
for you to fetch changes up to 7e74984e652fab200bc7319d7c3d90f6ae36be2e:
xfs_{db,repair}: add an explicit owner field to xfs_da_args (2024-07-29 17:01:06 -0700)
----------------------------------------------------------------
xfsprogs: set and validate dir/attr block owners [v30.9 04/28]
There are a couple of significatn changes that need to be made to the directory and xattr code before we can support online repairs of those data structures.
The first change is because online repair is designed to use libxfs to create a replacement dir/xattr structure in a temporary file, and use atomic extent swapping to commit the corrected structure. To avoid the performance hit of walking every block of the new structure to rewrite the owner number, we instead change libxfs to allow callers of the dir and xattr code the ability to set an explicit owner number to be written into the header fields of any new blocks that are created.
The second change is to update the dir/xattr code to actually *check* the owner number in each block that is read off the disk, since we don't currently do that.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (1):
xfs_{db,repair}: add an explicit owner field to xfs_da_args
db/namei.c | 1 +repair/phase6.c | 3 +++
2 files changed, 4 insertions(+)
[GIT PULL 05/23] xfsprogs: inode-related repair fixes
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 7e74984e652fab200bc7319d7c3d90f6ae36be2e:
xfs_{db,repair}: add an explicit owner field to xfs_da_args (2024-07-29 17:01:06 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/inode-repair-improvements-6.10_2024-07-29
for you to fetch changes up to ebf05a446c09336c08865dc29a6332be6ff8223c:
mkfs/repair: pin inodes that would otherwise overflow link count (2024-07-29 17:01:06 -0700)
----------------------------------------------------------------
xfsprogs: inode-related repair fixes [v30.9 05/28]
While doing QA of the online fsck code, I made a few observations:
First, nobody was checking that the di_onlink field is actually zero; Second, that allocating a temporary file for repairs can fail (and thus bring down the entire fs) if the inode cluster is corrupt; and Third, that file link counts do not pin at ~0U to prevent integer overflows.
This scattered patchset fixes those three problems.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (2):
libxfs: port the bumplink function from the kernel mkfs/repair: pin inodes that would otherwise overflow link count
include/xfs_inode.h | 2 ++ libxfs/util.c | 18 ++++++++++++++++++ mkfs/proto.c | 4 ++-- repair/incore_ino.c | 14 +++++++++----- repair/phase6.c | 10 +++++----- 5 files changed, 36 insertions(+), 12 deletions(-)
[GIT PULL 06/23] xfs_scrub: fixes to the repair code
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit ebf05a446c09336c08865dc29a6332be6ff8223c:
mkfs/repair: pin inodes that would otherwise overflow link count (2024-07-29 17:01:06 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-repair-fixes-6.10_2024-07-29
for you to fetch changes up to 4b959abc5f353123775973cd98c94d819cc9de79:
xfs_scrub: actually try to fix summary counters ahead of repairs (2024-07-29 17:01:06 -0700)
----------------------------------------------------------------xfs_scrub: fixes to the repair code [v30.9 06/28]
Now that we've landed the new kernel code, it's time to reorganize the xfs_scrub code that handles repairs. Clean up various naming warts and misleading error messages. Move the repair code to scrub/repair.c as the first step. Then, fix various issues in the repair code before we start reorganizing things.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (5):xfs_scrub: remove ALP_* flags namespacexfs_scrub: move repair functions to repair.cxfs_scrub: log when a repair was unnecessaryxfs_scrub: require primary superblock repairs to complete before proceedingxfs_scrub: actually try to fix summary counters ahead of repairs
scrub/phase1.c | 2 +- scrub/phase2.c | 3 +- scrub/phase3.c | 2 +- scrub/phase4.c | 22 ++++-- scrub/phase5.c | 2 +- scrub/phase7.c | 2 +- scrub/repair.c | 177 ++++++++++++++++++++++++++++++++++++++++++- scrub/repair.h | 16 +++- scrub/scrub.c | 204 +------------------------------------------------- scrub/scrub.h | 16 +--- scrub/scrub_private.h | 55 ++++++++++++++ 11 files changed, 269 insertions(+), 232 deletions(-)
create mode 100644 scrub/scrub_private.h
[GIT PULL 07/23] xfs_scrub: improve warnings about difficult repairs
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 4b959abc5f353123775973cd98c94d819cc9de79:
xfs_scrub: actually try to fix summary counters ahead of repairs (2024-07-29 17:01:06 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-better-repair-warnings-6.10_2024-07-29
for you to fetch changes up to bf15d7766e3dd63eda56f6b2f7976e529cd07575:
xfs_scrub: enable users to bump information messages to warnings (2024-07-29 17:01:07 -0700)
----------------------------------------------------------------xfs_scrub: improve warnings about difficult repairs [v30.9 07/28]
While I was poking through the QA results for xfs_scrub, I noticed that it doesn't warn the user when the primary and secondary realtime metadata are so out of whack that the chances of a successful repair are not so high. I decided that it was worth refactoring the scrub code a bit so that we could warn the user about these types of things, and ended up refactoring unnecessary helpers out of existence and fixing other reporting gaps.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (8):xfs_scrub: fix missing scrub coverage for broken inodesxfs_scrub: collapse trivial superblock scrub helpersxfs_scrub: get rid of trivial fs metadata scanner helpersxfs_scrub: split up the mustfix repairs and difficulty assessment functionsxfs_scrub: add missing repair types to the mustfix and difficulty assessmentxfs_scrub: any inconsistency in metadata should trigger difficulty warningsxfs_scrub: warn about difficult repairs to rt and quota metadataxfs_scrub: enable users to bump information messages to warnings
man/man8/xfs_scrub.8 | 19 +++++++++++++++++++ scrub/common.c | 2 ++ scrub/phase1.c | 2 +- scrub/phase2.c | 53 ++++++++++++++++++++++++++++++++-------------------- scrub/phase3.c | 21 +++++++++++++++++---- scrub/phase4.c | 9 +++++---- scrub/phase5.c | 15 +++++++-------- scrub/repair.c | 47 +++++++++++++++++++++++++++++++++++----------- scrub/repair.h | 10 +++++++--- scrub/scrub.c | 52 +-------------------------------------------------- scrub/scrub.h | 7 ++----- scrub/xfs_scrub.c | 45 +++++++++++++++++++++++++++++++++++++++++++- scrub/xfs_scrub.h | 1 + 13 files changed, 175 insertions(+), 108 deletions(-)
[GIT PULL 08/23] xfs_scrub: track data dependencies for repairs
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit bf15d7766e3dd63eda56f6b2f7976e529cd07575:
xfs_scrub: enable users to bump information messages to warnings (2024-07-29 17:01:07 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-repair-data-deps-6.10_2024-07-29
for you to fetch changes up to 81bfd0ad04a58892e2c153a22c361e7ff959f3fd:
xfs_scrub: remove unused action_list fields (2024-07-29 17:01:07 -0700)
----------------------------------------------------------------xfs_scrub: track data dependencies for repairs [v30.9 08/28]
Certain kinds of XFS metadata depend on the correctness of lower level metadata. For example, directory indexes depends on the directory data fork, which in turn depend on the directory inode to be correct. The current scrub code does not strictly preserve these dependencies if it has to defer a repair until phase 4, because phase 4 prioritizes repairs by type (corruption, then cross referencing, and then preening) and loses the ordering of in the previous phases. This leads to absurd things like trying to repair a directory before repairing its corrupted fork, which is absurd.
To solve this problem, introduce a repair ticket structure to track all the repairs pending for a principal object (inode, AG, etc). This reduces memory requirements if an object requires more than one type of repair and makes it very easy to track the data dependencies between sub-objects of a principal object. Repair dependencies between object types (e.g. bnobt before inodes) must still be encoded statically into phase 4.
A secondary benefit of this new ticket structure is that we can decide to attempt a repair of an object A that was flagged for a cross referencing error during the scan if a different object B depends on A but only B showed definitive signs of corruption.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (9):xfs_scrub: track repair items by principal, not by individual repairsxfs_scrub: use repair_item to direct repair activitiesxfs_scrub: remove action lists from phaseX codexfs_scrub: remove scrub_metadata_filexfs_scrub: boost the repair priority of dependencies of damaged itemsxfs_scrub: clean up repair_item_difficulty a littlexfs_scrub: check dependencies of a scrub type before repairingxfs_scrub: retry incomplete repairsxfs_scrub: remove unused action_list fields
libfrog/scrub.c | 1 + scrub/phase1.c | 9 +- scrub/phase2.c | 46 +++-- scrub/phase3.c | 77 ++++---- scrub/phase4.c | 17 +- scrub/phase5.c | 9 +- scrub/phase7.c | 9 +- scrub/repair.c | 530 ++++++++++++++++++++++++++++++++++---------------- scrub/repair.h | 47 +++-- scrub/scrub.c | 136 ++++++------- scrub/scrub.h | 108 ++++++++-- scrub/scrub_private.h | 37 ++++ 12 files changed, 664 insertions(+), 362 deletions(-)
[GIT PULL 09/23] xfs_scrub: use scrub_item to track check progress
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 81bfd0ad04a58892e2c153a22c361e7ff959f3fd:
xfs_scrub: remove unused action_list fields (2024-07-29 17:01:07 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-object-tracking-6.10_2024-07-29
for you to fetch changes up to 8dd67c8eccbe0e6b8dd7975ba53f9ccaf532aa9c:
xfs_scrub: hoist scrub retry loop to scrub_item_check_file (2024-07-29 17:01:08 -0700)
----------------------------------------------------------------xfs_scrub: use scrub_item to track check progress [v30.9 09/28]
Now that we've introduced tickets to track the status of repairs to a specific principal XFS object (fs, ag, file), use them to track the scrub state of those same objects. Ultimately, we want to make it easy to introduce vectorized repair, where we send a batch of repair requests to the kernel instead of making millions of ioctl calls. For now, however, we'll settle for easier bookkeepping.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (5):xfs_scrub: start tracking scrub state in scrub_itemxfs_scrub: remove enum check_outcomexfs_scrub: refactor scrub_meta_type out of existencexfs_scrub: hoist repair retry loop to repair_item_classxfs_scrub: hoist scrub retry loop to scrub_item_check_file
scrub/phase1.c | 3 +- scrub/phase2.c | 12 +- scrub/phase3.c | 41 ++----- scrub/phase4.c | 16 ++- scrub/phase5.c | 5 +- scrub/phase7.c | 5 +- scrub/repair.c | 71 ++++++----- scrub/scrub.c | 321 +++++++++++++++++++++++--------------------------- scrub/scrub.h | 40 +++++-- scrub/scrub_private.h | 14 +++ 10 files changed, 257 insertions(+), 271 deletions(-)
[GIT PULL 10/23] xfs_scrub: improve scheduling of repair items
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 8dd67c8eccbe0e6b8dd7975ba53f9ccaf532aa9c:
xfs_scrub: hoist scrub retry loop to scrub_item_check_file (2024-07-29 17:01:08 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-repair-scheduling-6.10_2024-07-29
for you to fetch changes up to 474ff27d466c053f1cd51024e6b0c5a741a2d4bd:
xfs_scrub: try to repair space metadata before file metadata (2024-07-29 17:01:08 -0700)
----------------------------------------------------------------xfs_scrub: improve scheduling of repair items [v30.9 10/28]
Currently, phase 4 of xfs_scrub uses per-AG repair item lists to schedule repair work across a thread pool. This scheme is suboptimal when most of the repairs involve a single AG because all the work gets dumped on a single pool thread.
Instead, we should create a thread pool with the same number of workers as CPUs, and dispatch individual repair tickets as separate work items to maximize parallelization.
However, we also need to ensure that repairs to space metadata and file metadata are kept in separate queues because file repairs generally depend on correctness of space metadata.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (4):
libfrog: enhance ptvar to support initializer functionsxfs_scrub: improve thread scheduling repair items during phase 4xfs_scrub: recheck entire metadata objects after corruption repairsxfs_scrub: try to repair space metadata before file metadata
libfrog/ptvar.c | 9 +- libfrog/ptvar.h | 4 +- scrub/counter.c | 2 +- scrub/descr.c | 2 +- scrub/phase1.c | 15 +++- scrub/phase2.c | 23 ++++- scrub/phase3.c | 106 ++++++++++++++-------- scrub/phase4.c | 244 +++++++++++++++++++++++++++++++++++++------------- scrub/phase7.c | 2 +- scrub/read_verify.c | 2 +- scrub/repair.c | 172 ++++++++++++++++++++++------------- scrub/repair.h | 37 ++++++-- scrub/scrub.c | 5 +- scrub/scrub.h | 10 +++ scrub/scrub_private.h | 2 + scrub/xfs_scrub.h | 3 +- 16 files changed, 455 insertions(+), 183 deletions(-)
[GIT PULL 11/23] xfs_scrub: detect deceptive filename extensions
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 474ff27d466c053f1cd51024e6b0c5a741a2d4bd:
xfs_scrub: try to repair space metadata before file metadata (2024-07-29 17:01:08 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-detect-deceptive-extensions-6.10_2024-07-29
for you to fetch changes up to 746ee95b71649b4ae515893ffa3bbe7b5e815d0d:
xfs_scrub: dump unicode points (2024-07-29 17:01:09 -0700)
----------------------------------------------------------------xfs_scrub: detect deceptive filename extensions [v30.9 11/28]
In early 2023, malware researchers disclosed a phishing attack that was targeted at people running Linux workstations. The attack vector involved the use of filenames containing what looked like a file extension but instead contained a lookalike for the full stop (".") and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect these types of attacks and warn the system administrator.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (13):xfs_scrub: use proper UChar string iteratorsxfs_scrub: hoist code that removes ignorable charactersxfs_scrub: add a couple of omitted invisible code pointsxfs_scrub: avoid potential UAF after freeing a duplicate name entryxfs_scrub: guard against libicu returning negative buffer lengthsxfs_scrub: hoist non-rendering character predicatexfs_scrub: store bad flags with the name entryxfs_scrub: rename UNICRASH_ZERO_WIDTH to UNICRASH_INVISIBLExfs_scrub: type-coerce the UNICRASH_* flagsxfs_scrub: reduce size of struct name_entryxfs_scrub: rename struct unicrash.normalizerxfs_scrub: report deceptive file extensionsxfs_scrub: dump unicode points
scrub/unicrash.c | 532 ++++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 426 insertions(+), 106 deletions(-)
[GIT PULL 12/23] xfs_scrub: move fstrim to a separate phase
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 746ee95b71649b4ae515893ffa3bbe7b5e815d0d:
xfs_scrub: dump unicode points (2024-07-29 17:01:09 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-fstrim-phase-6.10_2024-07-29
for you to fetch changes up to 5ccdd24dc9987b50333332b7381ff1a305e67ef7:
xfs_scrub: improve responsiveness while trimming the filesystem (2024-07-29 17:01:09 -0700)
----------------------------------------------------------------xfs_scrub: move fstrim to a separate phase [v30.9 12/28]
Back when I originally designed xfs_scrub, all filesystem metadata checks were complete by the end of phase 3, and phase 4 was where all the metadata repairs occurred. On the grounds that the filesystem should be fully consistent by then, I made a call to FITRIM at the end of phase 4 to discard empty space in the filesystem.
Unfortunately, that's no longer the case -- summary counters, link counts, and quota counters are not checked until phase 7. It's not safe to instruct the storage to unmap "empty" areas if we don't know where those empty areas are, so we need to create a phase 8 to trim the fs. While we're at it, make it more obvious that fstrim only gets to run if there are no unfixed corruptions and no other runtime errors have occurred.
Finally, reduce the latency impacts on the rest of the system by breaking up the fstrim work into a loop that targets only 16GB per call. This enables better progress reporting for interactive runs and cgroup based resource constraints for background runs.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (7):xfs_scrub: move FITRIM to phase 8xfs_scrub: ignore phase 8 if the user disabled fstrimxfs_scrub: collapse trim_filesystemxfs_scrub: fix the work estimation for phase 8xfs_scrub: report FITRIM errors properlyxfs_scrub: don't call FITRIM after runtime errorsxfs_scrub: improve responsiveness while trimming the filesystem
scrub/Makefile | 1 + scrub/phase4.c | 30 ++--------- scrub/phase8.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ scrub/vfs.c | 22 +++++--- scrub/vfs.h | 2 +- scrub/xfs_scrub.c | 11 +++- scrub/xfs_scrub.h | 3 ++ 7 files changed, 183 insertions(+), 37 deletions(-)
create mode 100644 scrub/phase8.c
[GIT PULL 13/23] xfs_scrub: use free space histograms to reduce fstrim runtime
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 5ccdd24dc9987b50333332b7381ff1a305e67ef7:
xfs_scrub: improve responsiveness while trimming the filesystem (2024-07-29 17:01:09 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-fstrim-minlen-freesp-histogram-6.10_2024-07-29
for you to fetch changes up to 34bed605490f936c3ead49e2e1cad78505260461:
xfs_scrub: tune fstrim minlen parameter based on free space histograms (2024-07-29 17:01:09 -0700)
----------------------------------------------------------------xfs_scrub: use free space histograms to reduce fstrim runtime [v30.9 13/28]
This patchset dramatically reduces the runtime of the FITRIM calls made during phase 8 of xfs_scrub. It turns out that phase 8 can really get bogged down if the free space contains a large number of very small extents. In these cases, the runtime can increase by an order of magnitude to free less than 1%% of the free space. This is not worth the time, since we're spending a lot of time to do very little work. The FITRIM ioctl allows us to specify a minimum extent length, so we can use statistical methods to compute a minlen parameter.
It turns out xfs_db/spaceman already have the code needed to create histograms of free space extent lengths. We add the ability to compute a CDF of the extent lengths, which make it easy to pick a minimum length corresponding to 99%% of the free space. In most cases, this results in dramatic reductions in phase 8 runtime. Hence, move the histogram code to libfrog, and wire up xfs_scrub, since phase 7 already walks the fsmap.
We also add a new -o suboption to xfs_scrub so that people who /do/ want to examine every free extent can do so.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (7):
libfrog: hoist free space histogram code
libfrog: print wider columns for free space histogram
libfrog: print cdf of free space bucketsxfs_scrub: don't close stdout when closing the progress barxfs_scrub: remove pointless spacemap.c argumentsxfs_scrub: collect free space histograms during phase 7xfs_scrub: tune fstrim minlen parameter based on free space histograms
db/freesp.c | 89 +++++------------ libfrog/Makefile | 2 + libfrog/histogram.c | 270 +++++++++++++++++++++++++++++++++++++++++++++++++++ libfrog/histogram.h | 77 +++++++++++++++ man/man8/xfs_scrub.8 | 16 +++ scrub/phase7.c | 47 ++++++++- scrub/phase8.c | 91 ++++++++++++++++- scrub/spacemap.c | 11 +-- scrub/vfs.c | 4 +- scrub/vfs.h | 2 +- scrub/xfs_scrub.c | 45 ++++++++- scrub/xfs_scrub.h | 16 +++ spaceman/freesp.c | 99 +++++++------------ 13 files changed, 619 insertions(+), 150 deletions(-)
create mode 100644 libfrog/histogram.c
create mode 100644 libfrog/histogram.h
[GIT PULL 14/23] xfs_scrub: tighten security of systemd services
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 34bed605490f936c3ead49e2e1cad78505260461:
xfs_scrub: tune fstrim minlen parameter based on free space histograms (2024-07-29 17:01:09 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-service-security-6.10_2024-07-29
for you to fetch changes up to 50411335572120153cc84d54213cd5ca9dd11b14:
xfs_scrub_all: tighten up the security on the background systemd service (2024-07-29 17:01:10 -0700)
----------------------------------------------------------------xfs_scrub: tighten security of systemd services [v30.9 14/28]
To reduce the risk of the online fsck service suffering some sort of catastrophic breach that results in attackers reconfiguring the running system, I embarked on a security audit of the systemd service files. The result should be that all elements of the background service (individual scrub jobs, the scrub_all initiator, and the failure reporting) run with as few privileges and within as strong of a sandbox as possible.
Granted, this does nothing about the potential for the /kernel/ screwing up, but at least we could prevent obvious container escapes.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (6):xfs_scrub: allow auxiliary pathnames for sandboxing xfs_scrub.service: reduce background CPU usage to less than one core if possiblexfs_scrub: use dynamic users when running as a systemd servicexfs_scrub: tighten up the security on the background systemd servicexfs_scrub_fail: tighten up the security on the background systemd servicexfs_scrub_all: tighten up the security on the background systemd service
man/man8/xfs_scrub.8 | 9 +++- scrub/Makefile | 7 ++- scrub/phase1.c | 4 +- scrub/system-xfs_scrub.slice | 30 +++++++++++++ scrub/vfs.c | 2 +- scrub/xfs_scrub.c | 11 +++-- scrub/xfs_scrub.h | 5 ++- scrub/[email protected] | 97 +++++++++++++++++++++++++++++++++++----- scrub/xfs_scrub_all.service.in | 66 +++++++++++++++++++++++++++ scrub/[email protected] | 59 ++++++++++++++++++++++++ 10 files changed, 270 insertions(+), 20 deletions(-)
create mode 100644 scrub/system-xfs_scrub.slice
[GIT PULL 15/23] xfs_scrub_all: automatic media scan service
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 50411335572120153cc84d54213cd5ca9dd11b14:
xfs_scrub_all: tighten up the security on the background systemd service (2024-07-29 17:01:10 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-media-scan-service-6.10_2024-07-29
for you to fetch changes up to e040916f649f71fb6a695dfcc6f6c9f576c5c4db:
xfs_scrub_all: failure reporting for the xfs_scrub_all job (2024-07-29 17:01:10 -0700)
----------------------------------------------------------------xfs_scrub_all: automatic media scan service [v30.9 15/28]
Now that we've completed the online fsck functionality, there are a few things that could be improved in the automatic service. Specifically, we would like to perform a more intensive metadata + media scan once per month, to give the user confidence that the filesystem isn't losing data silently. To accomplish this, enhance xfs_scrub_all to be able to trigger media scans. Next, add a duplicate set of system services that start the media scans automatically.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (6):xfs_scrub_all: only use the xfs_scrub@ systemd services in service modexfs_scrub_all: remove journalctl background processxfs_scrub_all: support metadata+media scans of all filesystemsxfs_scrub_all: enable periodic file data scrubs automaticallyxfs_scrub_all: trigger automatic media scans once per monthxfs_scrub_all: failure reporting for the xfs_scrub_all job
debian/rules | 3 +-
include/builddefs.in | 3 +
man/man8/Makefile | 7 +-
man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} | 20 +++-
scrub/Makefile | 21 +++-
scrub/[email protected] | 2 +-
scrub/xfs_scrub_all.cron.in | 2 +-
scrub/xfs_scrub_all.in | 126 ++++++++++++++++++-----
scrub/xfs_scrub_all.service.in | 9 +-
scrub/xfs_scrub_all_fail.service.in | 71 +++++++++++++
scrub/xfs_scrub_fail.in | 46 +++++++--
scrub/[email protected] | 2 +-
scrub/[email protected] | 100 ++++++++++++++++++
scrub/[email protected] | 76 ++++++++++++++
14 files changed, 443 insertions(+), 45 deletions(-)rename man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} (59%) create mode 100644 scrub/xfs_scrub_all_fail.service.in create mode 100644 scrub/[email protected] create mode 100644 scrub/[email protected]
[GIT PULL 16/23] xfs_scrub_all: improve systemd handling
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit e040916f649f71fb6a695dfcc6f6c9f576c5c4db:
xfs_scrub_all: failure reporting for the xfs_scrub_all job (2024-07-29 17:01:10 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-all-improve-systemd-handling-6.10_2024-07-29
for you to fetch changes up to e46249ec0b96ac5ced5b49928d3b3865b6d9546c:
xfs_scrub_all: implement retry and backoff for dbus calls (2024-07-29 17:01:10 -0700)
----------------------------------------------------------------xfs_scrub_all: improve systemd handling [v30.9 16/28]
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (5):xfs_scrub_all: encapsulate all the subprocess code in an objectxfs_scrub_all: encapsulate all the systemctl code in an objectxfs_scrub_all: add CLI option for easier debuggingxfs_scrub_all: convert systemctl calls to dbusxfs_scrub_all: implement retry and backoff for dbus calls
debian/control | 2 +-
scrub/xfs_scrub_all.in | 284 ++++++++++++++++++++++++++++++++++++-------------
2 files changed, 213 insertions(+), 73 deletions(-)
[GIT PULL 17/23] xfsprogs: improve extended attribute validation
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit e46249ec0b96ac5ced5b49928d3b3865b6d9546c:
xfs_scrub_all: implement retry and backoff for dbus calls (2024-07-29 17:01:10 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/improve-attr-validation-6.10_2024-07-29
for you to fetch changes up to 2823d8ed93da2bd3880abb52a58e91a920961e27:
xfs_repair: check for unknown flags in attr entries (2024-07-29 17:01:11 -0700)
----------------------------------------------------------------
xfsprogs: improve extended attribute validation [v13.8 17/28]
Prior to introducing parent pointer extended attributes, let's spend some time cleaning up the attr code and strengthening the validation that it performs on attrs coming in from the disk.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (6):xfs_scrub_all: fail fast on masked unitsxfs_scrub: automatic downgrades to dry-run mode in service modexfs_scrub: add an optimization-only modexfs_repair: check free space requirements before allowing upgradesxfs_repair: enforce one namespace bit per extended attributexfs_repair: check for unknown flags in attr entries
include/libxfs.h | 1 + libxfs/libxfs_api_defs.h | 1 + man/man8/xfs_scrub.8 | 6 ++- repair/attr_repair.c | 30 +++++++++++ repair/phase2.c | 134 +++++++++++++++++++++++++++++++++++++++++++++++ scrub/Makefile | 2 +- scrub/phase1.c | 13 +++++ scrub/phase4.c | 6 +++ scrub/repair.c | 37 ++++++++++++- scrub/repair.h | 2 + scrub/scrub.c | 4 +- scrub/xfs_scrub.c | 21 +++++++- scrub/xfs_scrub.h | 1 + scrub/xfs_scrub_all.in | 21 ++++++++ 14 files changed, 272 insertions(+), 7 deletions(-)
[GIT PULL 18/23] xfsprogs: Parent Pointers
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 2823d8ed93da2bd3880abb52a58e91a920961e27:
xfs_repair: check for unknown flags in attr entries (2024-07-29 17:01:11 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/pptrs-6.10_2024-07-29
for you to fetch changes up to b2677fa4f4009abf8de4c15960a2e97dd5370d41:
mkfs: enable formatting with parent pointers (2024-07-29 17:01:12 -0700)
----------------------------------------------------------------
xfsprogs: Parent Pointers [v13.8 18/28]
This is the latest parent pointer attributes for xfs. The goal of this patch set is to add a parent pointer attribute to each inode. The attribute name containing the parent inode, generation, and directory offset, while the attribute value contains the file name. This feature will enable future optimizations for online scrub, shrink, nfs handles, verity, or any other feature that could make use of quickly deriving an inodes path from the mount point.
Directory parent pointers are stored as namespaced extended attributes of a file. Because parent pointers are an indivisible tuple of (dirent_name, parent_ino, parent_gen) we cannot use the usual attr name lookup functions to find a parent pointer. This is solvable by introducing a new lookup mode that checks both the name and the value of the xattr.
Therefore, introduce this new name-value lookup mode that's gated on the XFS_ATTR_PARENT namespace. This requires the introduction of new opcodes for the extended attribute update log intent items, which actually means that parent pointers (itself an INCOMPAT feature) does not depend on the LOGGED_XATTRS log incompat feature bit.
To reduce collisions on the dirent names of parent pointers, introduce a new attr hash mode that is the dir2 namehash of the dirent name xor'd with the parent inode number.
At this point, Allison has moved on to other things, so I've merged her patchset into djwong-dev for merging.
Updates since v12 [djwong]:
Rebase on 6.9-rc and update the online fsck design document. Redesign the ondisk format to use the name-value lookups to get us back to the point where the attr is (dirent_name -> parent_ino/gen).
Updates since v11 [djwong]:
Rebase on 6.4-rc and make some tweaks and bugfixes to enable the repair prototypes. Merge with djwong-dev and make online repair actually work.
Updates since v10 [djwong]:
Merge in the ondisk format changes to get rid of the diroffset conflicts with the parent pointer repair code, rebase the entire series with the attr vlookup changes first, and merge all the other random fixes.
Updates since v9:
Reordered patches 2 and 3 to be 6 and 7
xfs: Add xfs_verify_pptr
moved parent pointer validators to xfs_parent
xfs: Add parent pointer ioctl
Extra validation checks for fs id
added missing release for the inode
use GFP_KERNEL flags for malloc/realloc reworked ioctl to use pptr listenty and flex array
NEW
xfs: don't remove the attr fork when parent pointers are enabled
NEW
directory lookups should return diroffsets too
NEW
xfs: move/add parent pointer validators to xfs_parent
Updates since v8:
xfs: parent pointer attribute creation
Fix xfs_parent_init to release log assist on alloc fail Add slab cache for xfs_parent_defer
Fix xfs_create to release after unlock
Add xfs_parent_start and xfs_parent_finish wrappers removed unused xfs_parent_name_irec and xfs_init_parent_name_irec
xfs: add parent attributes to link
Start/finish wrapper updates
Fix xfs_link to disallow reservationless quotas
xfs: add parent attributes to symlink
Fix xfs_symlink to release after unlock
Start/finish wrapper updates
xfs: remove parent pointers in unlink
Start/finish wrapper updates
Add missing parent free
xfs: Add parent pointers to rename
Start/finish wrapper updates
Fix rename to only grab logged xattr once
Fix xfs_rename to disallow reservationless quotas
Fix double unlock on dqattach fail
Move parent frees to out_release_wip
xfs: Add parent pointers to xfs_cross_rename
Hoist parent pointers into rename
Questions comments and feedback appreciated!
Thanks all!
Allison
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Allison Henderson (4):xfs_io: Add i, n and f flags to parent commandxfs_logprint: decode parent pointers in ATTRI items fully
mkfs: Add parent pointers during protofile creation
mkfs: enable formatting with parent pointers
Darrick J. Wong (20):
libxfs: create attr log item opcodes and formats for parent pointers xfs_{db,repair}: implement new attr hash value functionxfs_logprint: dump new attr log item fields
man: document the XFS_IOC_GETPARENTS ioctl
libfrog: report parent pointers to userspace
libfrog: add parent pointer support codexfs_io: adapt parent command to new parent pointer ioctlsxfs_spaceman: report file pathsxfs_scrub: use parent pointers when possible to report file operationsxfs_scrub: use parent pointers to report lost file dataxfs_db: report parent pointers in version commandxfs_db: report parent bit on xattrsxfs_db: report parent pointers embedded in xattrsxfs_db: obfuscate dirent and parent pointer names consistently
libxfs: export attr3_leaf_hdr_from_disk via libxfs_api_defs.hxfs_db: add a parents command to list the parents of a filexfs_db: make attr_set and attr_remove handle parent pointersxfs_db: add link and unlink expert commandsxfs_db: compute hashes of parent pointers
libxfs: create new files with attr forks if necessary
db/attr.c | 33 +- db/attrset.c | 202 +++++++++--- db/attrshort.c | 27 ++ db/field.c | 10 + db/field.h | 3 + db/hash.c | 44 ++- db/metadump.c | 322 ++++++++++++++++-- db/namei.c | 701 ++++++++++++++++++++++++++++++++++++++++ db/sb.c | 2 + include/handle.h | 1 + include/xfs_inode.h | 4 + io/parent.c | 541 +++++++++++-------------------- libfrog/Makefile | 2 + libfrog/fsgeom.c | 6 +- libfrog/getparents.c | 355 ++++++++++++++++++++ libfrog/getparents.h | 42 +++ libfrog/paths.c | 168 ++++++++++ libfrog/paths.h | 25 ++ libhandle/handle.c | 7 +- libxfs/defer_item.c | 52 ++- libxfs/init.c | 4 + libxfs/libxfs_api_defs.h | 19 ++ libxfs/util.c | 19 +- logprint/log_redo.c | 217 +++++++++++-- logprint/logprint.h | 5 +- man/man2/ioctl_xfs_getparents.2 | 212 ++++++++++++ man/man8/xfs_db.8 | 59 +++- man/man8/xfs_io.8 | 32 +- man/man8/xfs_spaceman.8 | 7 +- mkfs/lts_4.19.conf | 3 + mkfs/lts_5.10.conf | 3 + mkfs/lts_5.15.conf | 3 + mkfs/lts_5.4.conf | 3 + mkfs/lts_6.1.conf | 3 + mkfs/lts_6.6.conf | 3 + mkfs/proto.c | 62 +++- mkfs/xfs_mkfs.c | 45 ++- repair/attr_repair.c | 24 +- scrub/common.c | 41 ++- scrub/phase6.c | 75 ++++- spaceman/Makefile | 16 +- spaceman/file.c | 7 + spaceman/health.c | 53 ++- spaceman/space.h | 3 + 44 files changed, 2941 insertions(+), 524 deletions(-)
create mode 100644 libfrog/getparents.c create mode 100644 libfrog/getparents.h create mode 100644 man/man2/ioctl_xfs_getparents.2
[GIT PULL 19/23] xfsprogs: scrubbing for parent pointers
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit b2677fa4f4009abf8de4c15960a2e97dd5370d41:
mkfs: enable formatting with parent pointers (2024-07-29 17:01:12 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-pptrs-6.10_2024-07-29
for you to fetch changes up to 4b327cc2f5d03b772dd6d3352cbe28452cd41ef0:
man2: update ioctl_xfs_scrub_metadata.2 for parent pointers (2024-07-29 17:01:12 -0700)
----------------------------------------------------------------
xfsprogs: scrubbing for parent pointers [v13.8 19/28]
Teach online fsck to use parent pointers to assist in checking directories, parent pointers, extended attributes, and link counts.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (2):
xfs: create a blob array data structure
man2: update ioctl_xfs_scrub_metadata.2 for parent pointers
libxfs/Makefile | 2 + libxfs/xfblob.c | 147 ++++++++++++++++++++++++++++++++++++ libxfs/xfblob.h | 24 ++++++ libxfs/xfile.c | 11 +++ libxfs/xfile.h | 1 + man/man2/ioctl_xfs_scrub_metadata.2 | 20 ++++- 6 files changed, 201 insertions(+), 4 deletions(-)
create mode 100644 libxfs/xfblob.c
create mode 100644 libxfs/xfblob.h
[GIT PULL 20/23] xfsprogs: offline repair for parent pointers
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 4b327cc2f5d03b772dd6d3352cbe28452cd41ef0:
man2: update ioctl_xfs_scrub_metadata.2 for parent pointers (2024-07-29 17:01:12 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-pptrs-6.10_2024-07-29
for you to fetch changes up to 7ea215189a3cac45cb9323439318fcc3410727d4:
xfs_repair: wipe ondisk parent pointers when there are none (2024-07-29 17:01:13 -0700)
----------------------------------------------------------------
xfsprogs: offline repair for parent pointers [v13.8 20/28]
This series implements online checking and repair for directory parent pointer metadata. The checking half is fairly straightforward -- for each outgoing directory link (forward or backwards), grab the inode at the other end, and confirm that there's a corresponding link. If we can't grab an inode or lock it, we'll save that link for a slower loop that cycles all the locks, confirms the continued existence of the link, and rechecks the link if it's actually still there.
Repairs are a bit more involved -- for directories, we walk the entire filesystem to rebuild the dirents from parent pointer information. Parent pointer repairs do the same walk but rebuild the pptrs from the dirent information, but with the added twist that it duplicates all the xattrs so that it can use the atomic extent swapping code to commit the repairs atomically.
This introduces an added twist to the xattr repair code -- we use dirent hooks to detect a colliding update to the pptr data while we're not holding the ILOCKs; if one is detected, we restart the xattr salvaging process but this time hold all the ILOCKs until the end of the scan.
For offline repair, the phase6 directory connectivity scan generates an index of all the expected parent pointers in the filesystem. Then it walks each file and compares the parent pointers attached to that file against the index generated, and resyncs the results as necessary.
The last patch teaches xfs_scrub to report pathnames of files that are being repaired, when possible.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (12):xfs_db: remove some boilerplate from xfs_attr_setxfs_db: actually report errors from libxfs_attr_setxfs_repair: junk parent pointer attributes when filesystem doesn't support themxfs_repair: add parent pointers when messing with /lost+foundxfs_repair: junk duplicate hashtab entries when processing sf direntsxfs_repair: build a parent pointer indexxfs_repair: move the global dirent name store to a separate objectxfs_repair: deduplicate strings stored in string blobxfs_repair: check parent pointersxfs_repair: dump garbage parent pointer attributesxfs_repair: update ondisk parent pointer recordsxfs_repair: wipe ondisk parent pointers when there are none
db/attrset.c | 36 +- libxfs/libxfs_api_defs.h | 6 + libxfs/xfblob.c | 9 + libxfs/xfblob.h | 2 + repair/Makefile | 6 + repair/attr_repair.c | 30 ++ repair/listxattr.c | 271 ++++++++++ repair/listxattr.h | 15 + repair/phase6.c | 121 ++++- repair/pptr.c | 1331 ++++++++++++++++++++++++++++++++++++++++++++++ repair/pptr.h | 17 + repair/strblobs.c | 211 ++++++++ repair/strblobs.h | 24 + 13 files changed, 2069 insertions(+), 10 deletions(-)
create mode 100644 repair/listxattr.c
create mode 100644 repair/listxattr.h
create mode 100644 repair/pptr.c
create mode 100644 repair/pptr.h
create mode 100644 repair/strblobs.c
create mode 100644 repair/strblobs.h
[GIT PULL 21/23] xfsprogs: detect and correct directory tree problems
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 7ea215189a3cac45cb9323439318fcc3410727d4:
xfs_repair: wipe ondisk parent pointers when there are none (2024-07-29 17:01:13 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-directory-tree-6.10_2024-07-29
for you to fetch changes up to 5a30504f0c60e10dc0cecd201c5afc18083fd0ac:
xfs_scrub: defer phase5 file scans if dirloop fails (2024-07-29 17:01:13 -0700)
----------------------------------------------------------------
xfsprogs: detect and correct directory tree problems [v13.8 21/28]
Historically, checking the tree-ness of the directory tree structure has not been complete. Cycles of subdirectories break the tree properties, as do subdirectories with multiple parents. It's easy enough for DFS to detect problems as long as one of the participants is reachable from the root, but this technique cannot find unconnected cycles.
Directory parent pointers change that, because we can discover all of these problems from a simple walk from a subdirectory towards the root. For each child we start with, if the walk terminates without reaching the root, we know the path is disconnected and ought to be attached to the lost and found. If we find ourselves, we know this is a cycle and can delete an incoming edge. If we find multiple paths to the root, we know to delete an incoming edge.
Even better, once we've finished walking paths, we've identified the good ones and know which other path(s) to remove.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (5):
libfrog: add directory tree structure scrubber to scrub libraryxfs_spaceman: report directory tree corruption in the health informationxfs_scrub: fix erroring out of check_inode_namesxfs_scrub: detect and repair directory tree corruptionsxfs_scrub: defer phase5 file scans if dirloop fails
libfrog/scrub.c | 5 + man/man2/ioctl_xfs_bulkstat.2 | 3 + man/man2/ioctl_xfs_fsbulkstat.2 | 3 + man/man2/ioctl_xfs_scrub_metadata.2 | 14 ++ scrub/phase5.c | 271 ++++++++++++++++++++++++++++++++++-- scrub/repair.c | 13 ++ scrub/repair.h | 2 + spaceman/health.c | 4 + 8 files changed, 301 insertions(+), 14 deletions(-)
[GIT PULL 22/23] xfs_scrub: vectorize kernel calls
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit 5a30504f0c60e10dc0cecd201c5afc18083fd0ac:
xfs_scrub: defer phase5 file scans if dirloop fails (2024-07-29 17:01:13 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/vectorized-scrub-6.10_2024-07-29
for you to fetch changes up to df914edeeb1e1919831192951d657cfc73b46418:
xfs_scrub: try spot repairs of metadata items to make scrub progress (2024-07-29 17:01:13 -0700)
----------------------------------------------------------------xfs_scrub: vectorize kernel calls [v30.9 22/28]
Create a vectorized version of the metadata scrub and repair ioctl, and adapt xfs_scrub to use that. This is an experiment to measure overhead and to try refactoring xfs_scrub.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (10):
man: document vectored scrub mode
libfrog: support vectored scrubxfs_io: support vectored scrubxfs_scrub: split the scrub epilogue code into a separate functionxfs_scrub: split the repair epilogue code into a separate functionxfs_scrub: convert scrub and repair epilogues to use xfs_scrub_vecxfs_scrub: vectorize scrub callsxfs_scrub: vectorize repair callsxfs_scrub: use scrub barriers to reduce kernel callsxfs_scrub: try spot repairs of metadata items to make scrub progress
io/scrub.c | 368 +++++++++++++++++++++++++++++++---- libfrog/fsgeom.h | 6 + libfrog/scrub.c | 137 +++++++++++++ libfrog/scrub.h | 35 ++++ man/man2/ioctl_xfs_scrubv_metadata.2 | 171 ++++++++++++++++ man/man8/xfs_io.8 | 51 +++++ scrub/phase1.c | 2 + scrub/phase2.c | 93 +++++++-- scrub/phase3.c | 84 ++++++-- scrub/repair.c | 355 ++++++++++++++++++++------------- scrub/scrub.c | 360 +++++++++++++++++++++++++--------- scrub/scrub.h | 19 ++ scrub/scrub_private.h | 55 ++++-- scrub/xfs_scrub.c | 1 + 14 files changed, 1431 insertions(+), 306 deletions(-)
create mode 100644 man/man2/ioctl_xfs_scrubv_metadata.2
[GIT PULL 23/23] xfs_repair: fixes for kernel 6.10
In reply to Darrick J. WongHi Carlos,
Please pull this branch with changes for xfsprogs for 6.10-rc1.
As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.
The following changes since commit df914edeeb1e1919831192951d657cfc73b46418:
xfs_scrub: try spot repairs of metadata items to make scrub progress (2024-07-29 17:01:13 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-fixes-6.10_2024-07-29
for you to fetch changes up to 5a43a00432ebe9ab8b54155703a9eb9e1a1dd4ec:
xfs_repair: allow symlinks with short remote targets (2024-07-29 17:01:13 -0700)
----------------------------------------------------------------xfs_repair: fixes for kernel 6.10 [v30.9 23/28]
Fix some incorrect validation problems in xfs_repair.
This has been running on the djcloud for months with no problems. Enjoy!
----------------------------------------------------------------
Darrick J. Wong (1):xfs_repair: allow symlinks with short remote targets
repair/dinode.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
Quoted 3 lines · Darrick J. Wong
Hi Carlos,
Please pull these fully reviewed patchsets for xfsprogs 6.10.
All pulled, thanks!
Carlos
Quoted 3 lines
--D