Skip to contentSkip to conversation

xfs: online repair, part 1 is done

GIT PULLBOMB v30.3 Darrick J. Wong 1 participant 17 messages Apr 16, 2024 VFS

lore
Share & tools
lore
#0 lore raw

Hi Chandan,

Please accept these pull requests for all the fully reviewed patchsets that I have in my development tree. The first fourteen pulls will complete online repair part 1; the last two are from the parent pointers patchset, which is part 2. I will be resending the unfinished patchsets of part 2 later on to try to wrap up the last few patches that have not completed review.

--D

#1 lore raw

[GIT PULL 01/16] xfs: improve log incompat feature handling

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 0bbac3facb5d6cc0171c45c9873a2dc96bea9680:

Linux 6.9-rc4 (2024-04-14 13:38:39 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/log-incompat-permissions-6.10_2024-04-15

for you to fetch changes up to 5302a5c8beb21d01b7b8d92cc73b6871bc27d7bf:

xfs: only clear log incompat flags at clean unmount (2024-04-15 14:54:06 -0700)

----------------------------------------------------------------
xfs: improve log incompat feature handling [v30.3 01/16]

This patchset improves the performance of log incompat feature bit handling by making a few changes to how the filesystem handles them. First, we now only clear the bits during a clean unmount to reduce calls to the (expensive) upgrade function to once per bit per mount. Second, we now only allow incompat feature upgrades for sysadmins or if the sysadmin explicitly allows it via mount option. Currently the only log incompat user is logged xattrs, which requires CONFIG_XFS_DEBUG=y, so there should be no user visible impact to this change.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (5):
xfs: pass xfs_buf lookup flags to xfs_*read_agi
xfs: fix an AGI lock acquisition ordering problem in xrep_dinode_findmode
xfs: fix potential AGI <-> ILOCK ABBA deadlock in xrep_dinode_findmode_walk_directory
xfs: fix error bailout in xrep_abt_build_new_trees
xfs: only clear log incompat flags at clean unmount

.../filesystems/xfs/xfs-online-fsck-design.rst     |  3 --
fs/xfs/libxfs/xfs_ag.c                             |  8 ++--
fs/xfs/libxfs/xfs_ialloc.c                         | 16 ++++---
fs/xfs/libxfs/xfs_ialloc.h                         |  5 ++-
fs/xfs/libxfs/xfs_ialloc_btree.c                   |  4 +-
fs/xfs/scrub/alloc_repair.c                        |  2 +-
fs/xfs/scrub/common.c                              |  4 +-
fs/xfs/scrub/fscounters.c                          |  2 +-
fs/xfs/scrub/inode_repair.c                        | 50 +++++++++++++++++++++-
fs/xfs/scrub/iscan.c                               | 36 +++++++++++++++-
fs/xfs/scrub/iscan.h                               | 15 +++++++
fs/xfs/scrub/repair.c                              |  6 +--
fs/xfs/scrub/trace.h                               | 10 ++++-
fs/xfs/xfs_inode.c                                 |  8 ++--
fs/xfs/xfs_iwalk.c                                 |  4 +-
fs/xfs/xfs_log.c                                   | 28 +-----------
fs/xfs/xfs_log.h                                   |  2 -
fs/xfs/xfs_log_priv.h                              |  3 --
fs/xfs/xfs_log_recover.c                           | 19 +-------
fs/xfs/xfs_mount.c                                 |  8 +++-
fs/xfs/xfs_mount.h                                 |  6 ++-
fs/xfs/xfs_xattr.c                                 | 42 +++---------------
22 files changed, 160 insertions(+), 121 deletions(-)
#2 lore raw

[GIT PULL 02/16] xfs: refactorings for atomic file content exchanges

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 5302a5c8beb21d01b7b8d92cc73b6871bc27d7bf:

xfs: only clear log incompat flags at clean unmount (2024-04-15 14:54:06 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/file-exchange-refactorings-6.10_2024-04-15

for you to fetch changes up to 15f78aa3eb07645e7bef15a53b4ae1c757907d2c:

xfs: constify xfs_bmap_is_written_extent (2024-04-15 14:54:12 -0700)

----------------------------------------------------------------
xfs: refactorings for atomic file content exchanges [v30.3 02/16]

This series applies various cleanups and refactorings to file IO handling code ahead of the main series to implement atomic file content exchanges.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (7):
xfs: move inode lease breaking functions to xfs_inode.c
xfs: move xfs_iops.c declarations out of xfs_inode.h
xfs: declare xfs_file.c symbols in xfs_file.h
xfs: create a new helper to return a file's allocation unit
xfs: hoist multi-fsb allocation unit detection to a helper
xfs: refactor non-power-of-two alignment checks
xfs: constify xfs_bmap_is_written_extent

fs/xfs/libxfs/xfs_bmap.h |  2 +-
fs/xfs/xfs_bmap_util.c   |  4 +--
fs/xfs/xfs_file.c        | 88 ++++--------------------------------------------
fs/xfs/xfs_file.h        | 15 +++++++++
fs/xfs/xfs_inode.c       | 75 +++++++++++++++++++++++++++++++++++++++++
fs/xfs/xfs_inode.h       | 16 +++++----
fs/xfs/xfs_ioctl.c       |  1 +
fs/xfs/xfs_iops.c        |  1 +
fs/xfs/xfs_iops.h        |  7 ++--
fs/xfs/xfs_linux.h       |  5 +++
10 files changed, 121 insertions(+), 93 deletions(-)

create mode 100644 fs/xfs/xfs_file.h

#3 lore raw

[GIT PULL 03/16] xfs: atomic file content exchanges

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 15f78aa3eb07645e7bef15a53b4ae1c757907d2c:

xfs: constify xfs_bmap_is_written_extent (2024-04-15 14:54:12 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/atomic-file-updates-6.10_2024-04-15

for you to fetch changes up to 0730e8d8ba1d1507f1d7fd719e1f835ce69961fe:

xfs: enable logged file mapping exchange feature (2024-04-15 14:54:26 -0700)

----------------------------------------------------------------
xfs: atomic file content exchanges [v30.3 03/16]

This series creates a new XFS_IOC_EXCHANGE_RANGE ioctl to exchange ranges of bytes between two files atomically.

This new functionality enables data storage programs to stage and commit file updates such that reader programs will see either the old contents or the new contents in their entirety, with no chance of torn writes. A successful call completion guarantees that the new contents will be seen even if the system fails.

The ability to exchange file fork mappings between files in this manner is critical to supporting online filesystem repair, which is built upon the strategy of constructing a clean copy of a damaged structure and committing the new structure into the metadata file atomically. The ioctls exist to facilitate testing of the new functionality and to enable future application program designs.

User programs will be able to update files atomically by opening an O_TMPFILE, reflinking the source file to it, making whatever updates they want to make, and exchange the relevant ranges of the temp file with the original file. If the updates are aligned with the file block size, a new (since v2) flag provides for exchanging only the written areas. Note that application software must quiesce writes to the file while it stages an atomic update. This will be addressed by a subsequent series.

This mechanism solves the clunkiness of two existing atomic file update
mechanisms: for O_TRUNC + rewrite, this eliminates the brief period where other programs can see an empty file. For create tempfile + rename, the need to copy file attributes and extended attributes for each file update is eliminated.

However, this method introduces its own awkwardness -- any program initiating an exchange now needs to have a way to signal to other programs that the file contents have changed. For file access mediated via read and write, fanotify or inotify are probably sufficient. For mmaped files, that may not be fast enough.

Here is the proposed manual page:

IOCTL-XFS-EXCHANGE-RANGE(2System Calls ManuIOCTL-XFS-EXCHANGE-RANGE(2)

NAME
ioctl_xfs_exchange_range - exchange the contents of parts of two files

SYNOPSIS
#include <sys/ioctl.h>
#include <xfs/xfs_fs.h>

int ioctl(int file2_fd, XFS_IOC_EXCHANGE_RANGE, struct xfs_ex‐ change_range *arg);

DESCRIPTION
Given a range of bytes in a first file file1_fd and a second range of bytes in a second file file2_fd, this ioctl(2) ex‐ changes the contents of the two ranges.

Exchanges are atomic with regards to concurrent file opera‐ tions. Implementations must guarantee that readers see either the old contents or the new contents in their entirety, even if the system fails.

The system call parameters are conveyed in structures of the following form:

struct xfs_exchange_range {

__s32    file1_fd;
__u32    pad;
__u64    file1_offset;
__u64    file2_offset;
__u64    length;
__u64    flags;

};

The field pad must be zero.

The fields file1_fd, file1_offset, and length define the first range of bytes to be exchanged.

The fields file2_fd, file2_offset, and length define the second range of bytes to be exchanged.

Both files must be from the same filesystem mount. If the two file descriptors represent the same file, the byte ranges must
not overlap. Most disk-based filesystems require that the
starts of both ranges must be aligned to the file block size. If this is the case, the ends of the ranges must also be so aligned unless the XFS_EXCHANGE_RANGE_TO_EOF flag is set.

The field flags control the behavior of the exchange operation.

XFS_EXCHANGE_RANGE_TO_EOF
Ignore the length parameter. All bytes in file1_fd from file1_offset to EOF are moved to file2_fd, and file2's size is set to (file2_offset+(file1_length-
file1_offset)). Meanwhile, all bytes in file2 from
file2_offset to EOF are moved to file1 and file1's
size is set to (file1_offset+(file2_length-
file2_offset)).

XFS_EXCHANGE_RANGE_DSYNC
Ensure that all modified in-core data in both file ranges and all metadata updates pertaining to the exchange operation are flushed to persistent storage before the call returns. Opening either file de‐ scriptor with O_SYNC or O_DSYNC will have the same effect.

XFS_EXCHANGE_RANGE_FILE1_WRITTEN
Only exchange sub-ranges of file1_fd that are known

to contain data  written  by  application  software.
Each  sub-range  may  be  expanded (both upwards and

downwards) to align with the file allocation unit. For files on the data device, this is one filesystem block. For files on the realtime device, this is the realtime extent size. This facility can be used to implement fast atomic scatter-gather writes of any complexity for software-defined storage targets if all writes are aligned to the file allocation unit.

XFS_EXCHANGE_RANGE_DRY_RUN
Check the parameters and the feasibility of the op‐ eration, but do not change anything.

RETURN VALUE
On error, -1 is returned, and errno is set to indicate the er‐ ror.

ERRORS
Error codes can be one of, but are not limited to, the follow‐ ing:

EBADF file1_fd is not open for reading and writing or is open for append-only writes; or file2_fd is not open for reading and writing or is open for append-only writes.

EINVAL The parameters are not correct for these files. This
error can also appear if either file descriptor repre‐ sents a device, FIFO, or socket. Disk filesystems gen‐ erally require the offset and length arguments to be aligned to the fundamental block sizes of both files.

EIO An I/O error occurred.

EISDIR One of the files is a directory.

ENOMEM The kernel was unable to allocate sufficient memory to perform the operation.

ENOSPC There is not enough free space in the filesystem ex‐ change the contents safely.

EOPNOTSUPP
The filesystem does not support exchanging bytes between the two files.

EPERM file1_fd or file2_fd are immutable.

ETXTBSY
One of the files is a swap file.

EUCLEAN
The filesystem is corrupt.

EXDEV file1_fd and file2_fd are not on the same mounted filesystem.

CONFORMING TO
This API is XFS-specific.

USE CASES
Several use cases are imagined for this system call. In all
cases, application software must coordinate updates to the file because the exchange is performed unconditionally.

The first is a data storage program that wants to commit non- contiguous updates to a file atomically and coordinates write access to that file. This can be done by creating a temporary file, calling FICLONE(2) to share the contents, and staging the updates into the temporary file. The FULL_FILES flag is recom‐ mended for this purpose. The temporary file can be deleted or punched out afterwards.

An example program might look like this:

int fd = open("/some/file", O_RDWR);
int temp_fd = open("/some", O_TMPFILE | O_RDWR);

ioctl(temp_fd, FICLONE, fd);

/* append 1MB of records */
lseek(temp_fd, 0, SEEK_END);
write(temp_fd, data1, 1000000);

/* update record index */
pwrite(temp_fd, data1, 600, 98765);
pwrite(temp_fd, data2, 320, 54321);
pwrite(temp_fd, data2, 15, 0);

/* commit the entire update */
struct xfs_exchange_range args = {
.file1_fd = temp_fd,
.flags = XFS_EXCHANGE_RANGE_TO_EOF,
};

ioctl(fd, XFS_IOC_EXCHANGE_RANGE, &args);

The second is a software-defined storage host (e.g. a disk jukebox) which implements an atomic scatter-gather write com‐ mand. Provided the exported disk's logical block size matches the file's allocation unit size, this can be done by creating a temporary file and writing the data at the appropriate offsets. It is recommended that the temporary file be truncated to the size of the regular file before any writes are staged to the temporary file to avoid issues with zeroing during EOF exten‐ sion. Use this call with the FILE1_WRITTEN flag to exchange only the file allocation units involved in the emulated de‐ vice's write command. The temporary file should be truncated or punched out completely before being reused to stage another write.

An example program might look like this:

int fd = open("/some/file", O_RDWR);
int temp_fd = open("/some", O_TMPFILE | O_RDWR);
struct stat sb;
int blksz;

fstat(fd, &sb);
blksz = sb.st_blksize;

/* land scatter gather writes between 100fsb and 500fsb */ pwrite(temp_fd, data1, blksz * 2, blksz * 100); pwrite(temp_fd, data2, blksz * 20, blksz * 480); pwrite(temp_fd, data3, blksz * 7, blksz * 257);

/* commit the entire update */
struct xfs_exchange_range args = {
.file1_fd = temp_fd,
.file1_offset = blksz * 100,
.file2_offset = blksz * 100,
.length = blksz * 400,
.flags = XFS_EXCHANGE_RANGE_FILE1_WRITTEN |
XFS_EXCHANGE_RANGE_FILE1_DSYNC,
};

ioctl(fd, XFS_IOC_EXCHANGE_RANGE, &args);

NOTES
Some filesystems may limit the amount of data or the number of extents that can be exchanged in a single call.

SEE ALSO
ioctl(2)

XFS 2024-02-10 IOCTL-XFS-EXCHANGE-RANGE(2)

The reference implementation in XFS creates a new log incompat feature and log intent items to track high level progress of swapping ranges of two files and finish interrupted work if the system goes down. Sample code can be found in the corresponding changes to xfs_io to exercise the use case mentioned above.

Note that this function is /not/ the O_DIRECT atomic untorn file writes concept that has also been floating around for years. It is also not the RWF_ATOMIC patchset that has been shared. This RFC is constructed entirely in software, which means that there are no limitations other than the general filesystem limits.

As a side note, the original motivation behind the kernel functionality is online repair of file-based metadata. The atomic file content exchange is implemented as an atomic exchange of file fork mappings, which means that we can implement online reconstruction of extended attributes and directories by building a new one in another inode and exchanging the contents.

Subsequent patchsets adapt the online filesystem repair code to use atomic file exchanges. This enables repair functions to construct a clean copy of a directory, xattr information, symbolic links, realtime bitmaps, and realtime summary information in a temporary inode. If this completes successfully, the new contents can be committed atomically into the inode being repaired. This is essential to avoid making corruption problems worse if the system goes down in the middle of running repair.

For userspace, this series also includes the userspace pieces needed to test the new functionality, and a sample implementation of atomic file updates.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (15):
vfs: export remap and write check helpers
xfs: introduce new file range exchange ioctl
xfs: create a incompat flag for atomic file mapping exchanges
xfs: introduce a file mapping exchange log intent item
xfs: create deferred log items for file mapping exchanges
xfs: bind together the front and back ends of the file range exchange code
xfs: add error injection to test file mapping exchange recovery
xfs: condense extended attributes after a mapping exchange operation
xfs: condense directories after a mapping exchange operation
xfs: condense symbolic links after a mapping exchange operation
xfs: make file range exchange support realtime files
xfs: support non-power-of-two rtextsize with exchange-range
xfs: capture inode generation numbers in the ondisk exchmaps log item
docs: update swapext -> exchmaps language
xfs: enable logged file mapping exchange feature

.../filesystems/xfs/xfs-online-fsck-design.rst     |  259 ++--
fs/read_write.c                                    |    1 +
fs/remap_range.c                                   |    4 +-
fs/xfs/Makefile                                    |    3 +
fs/xfs/libxfs/xfs_defer.c                          |    6 +
fs/xfs/libxfs/xfs_defer.h                          |    2 +-
fs/xfs/libxfs/xfs_errortag.h                       |    4 +-
fs/xfs/libxfs/xfs_exchmaps.c                       | 1237 ++++++++++++++++++++
fs/xfs/libxfs/xfs_exchmaps.h                       |  123 ++
fs/xfs/libxfs/xfs_format.h                         |   26 +-
fs/xfs/libxfs/xfs_fs.h                             |   42 +
fs/xfs/libxfs/xfs_log_format.h                     |   66 +-
fs/xfs/libxfs/xfs_log_recover.h                    |    4 +
fs/xfs/libxfs/xfs_sb.c                             |    5 +
fs/xfs/libxfs/xfs_symlink_remote.c                 |   47 +
fs/xfs/libxfs/xfs_symlink_remote.h                 |    1 +
fs/xfs/libxfs/xfs_trans_space.h                    |    4 +
fs/xfs/xfs_error.c                                 |    3 +
fs/xfs/xfs_exchmaps_item.c                         |  614 ++++++++++
fs/xfs/xfs_exchmaps_item.h                         |   64 +
fs/xfs/xfs_exchrange.c                             |  804 +++++++++++++
fs/xfs/xfs_exchrange.h                             |   38 +
fs/xfs/xfs_ioctl.c                                 |    4 +
fs/xfs/xfs_log_recover.c                           |   33 +
fs/xfs/xfs_mount.h                                 |    2 +
fs/xfs/xfs_super.c                                 |   23 +
fs/xfs/xfs_symlink.c                               |   49 +-
fs/xfs/xfs_trace.c                                 |    2 +
fs/xfs/xfs_trace.h                                 |  327 ++++++
include/linux/fs.h                                 |    1 +
30 files changed, 3613 insertions(+), 185 deletions(-)

create mode 100644 fs/xfs/libxfs/xfs_exchmaps.c
create mode 100644 fs/xfs/libxfs/xfs_exchmaps.h
create mode 100644 fs/xfs/xfs_exchmaps_item.c
create mode 100644 fs/xfs/xfs_exchmaps_item.h
create mode 100644 fs/xfs/xfs_exchrange.c
create mode 100644 fs/xfs/xfs_exchrange.h

#4 lore raw

[GIT PULL 04/16] xfs: create temporary files for online repair

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 0730e8d8ba1d1507f1d7fd719e1f835ce69961fe:

xfs: enable logged file mapping exchange feature (2024-04-15 14:54:26 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-tempfiles-6.10_2024-04-15

for you to fetch changes up to 5befb047b9f4de1747bf48c63cab997a97e0088b:

xfs: add the ability to reap entire inode forks (2024-04-15 14:58:49 -0700)

----------------------------------------------------------------
xfs: create temporary files for online repair [v30.3 04/16]

As mentioned earlier, the repair strategy for file-based metadata is to build a new copy in a temporary file and swap the file fork mappings with the metadata inode. We've built the atomic extent swap facility, so now we need to build a facility for handling private temporary files.

The first step is to teach the filesystem to ignore the temporary files. We'll mark them as PRIVATE in the VFS so that the kernel security modules will leave it alone. The second step is to add the online repair code the ability to create a temporary file and reap extents from the temporary file after the extent swap.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (4):
xfs: hide private inodes from bulkstat and handle functions
xfs: create temporary files and directories for online repair
xfs: refactor live buffer invalidation for repairs
xfs: add the ability to reap entire inode forks

fs/xfs/Makefile         |   1 +
fs/xfs/scrub/parent.c   |   2 +-
fs/xfs/scrub/reap.c     | 445 +++++++++++++++++++++++++++++++++++++++++++++---
fs/xfs/scrub/reap.h     |  21 +++
fs/xfs/scrub/scrub.c    |   3 +
fs/xfs/scrub/scrub.h    |   4 +
fs/xfs/scrub/tempfile.c | 251 +++++++++++++++++++++++++++
fs/xfs/scrub/tempfile.h |  28 +++
fs/xfs/scrub/trace.h    |  96 +++++++++++
fs/xfs/xfs_export.c     |   2 +-
fs/xfs/xfs_inode.c      |   3 +-
fs/xfs/xfs_inode.h      |   2 +
fs/xfs/xfs_iops.c       |   3 +
fs/xfs/xfs_itable.c     |   8 +
14 files changed, 843 insertions(+), 26 deletions(-)

create mode 100644 fs/xfs/scrub/tempfile.c
create mode 100644 fs/xfs/scrub/tempfile.h

#5 lore raw

[GIT PULL 05/16] xfs: online repair of realtime summaries

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 5befb047b9f4de1747bf48c63cab997a97e0088b:

xfs: add the ability to reap entire inode forks (2024-04-15 14:58:49 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-rtsummary-6.10_2024-04-15

for you to fetch changes up to abf039e2e4afde98e448253f9a7ecc784a87924d:

xfs: online repair of realtime summaries (2024-04-15 14:58:49 -0700)

----------------------------------------------------------------
xfs: online repair of realtime summaries [v30.3 05/16]

We now have all the infrastructure we need to repair file metadata. We'll begin with the realtime summary file, because it is the least complex data structure. To support this we need to add three more pieces to the temporary file code from the previous patchset -- preallocating space in the temp file, formatting metadata into that space and writing the blocks to disk, and swapping the fork mappings atomically.

After that, the actual reconstruction of the realtime summary information is pretty simple, since we can simply write the incore copy computed by the rtsummary scrubber to the temporary file, swap the contents, and reap the old blocks.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: support preallocating and copying content into temporary files
xfs: teach the tempfile to set up atomic file content exchanges
xfs: online repair of realtime summaries

fs/xfs/Makefile                 |   1 +
fs/xfs/scrub/common.c           |   1 +
fs/xfs/scrub/repair.h           |   3 +
fs/xfs/scrub/rtsummary.c        |  33 ++--
fs/xfs/scrub/rtsummary.h        |  37 ++++
fs/xfs/scrub/rtsummary_repair.c | 177 ++++++++++++++++++
fs/xfs/scrub/scrub.c            |  11 +-
fs/xfs/scrub/scrub.h            |   7 +
fs/xfs/scrub/tempexch.h         |  21 +++
fs/xfs/scrub/tempfile.c         | 388 ++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/tempfile.h         |  15 ++
fs/xfs/scrub/trace.h            |  40 +++++
12 files changed, 715 insertions(+), 19 deletions(-)

create mode 100644 fs/xfs/scrub/rtsummary.h create mode 100644 fs/xfs/scrub/rtsummary_repair.c create mode 100644 fs/xfs/scrub/tempexch.h

#6 lore raw

[GIT PULL 06/16] xfs: set and validate dir/attr block owners

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit abf039e2e4afde98e448253f9a7ecc784a87924d:

xfs: online repair of realtime summaries (2024-04-15 14:58:49 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/dirattr-validate-owners-6.10_2024-04-15

for you to fetch changes up to fe6c9f8e48e0dcbfc3dba17edd88490c8579b34b:

xfs: validate explicit directory free block owners (2024-04-15 14:58:52 -0700)

----------------------------------------------------------------
xfs: set and validate dir/attr block owners [v30.3 06/16]

There are a couple of significant changes that need to be made to the directory and xattr code before we can support online repairs of those data structures.

The first change is because online repair is designed to use libxfs to create a replacement dir/xattr structure in a temporary file, and use atomic extent swapping to commit the corrected structure. To avoid the performance hit of walking every block of the new structure to rewrite the owner number before the swap, we instead change libxfs to allow callers of the dir and xattr code the ability to set an explicit owner number to be written into the header fields of any new blocks that are created. For regular operation this will be the directory inode number.

The second change is to update the dir/xattr code to actually *check* the owner number in each block that is read off the disk, since we don't currently do that.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (10):
xfs: add an explicit owner field to xfs_da_args
xfs: use the xfs_da_args owner field to set new dir/attr block owner
xfs: reduce indenting in xfs_attr_node_list
xfs: validate attr leaf buffer owners
xfs: validate attr remote value buffer owners
xfs: validate dabtree node buffer owners
xfs: validate directory leaf buffer owners
xfs: validate explicit directory data buffer owners
xfs: validate explicit directory block buffer owners
xfs: validate explicit directory free block owners

fs/xfs/libxfs/xfs_attr.c        |  14 ++--
fs/xfs/libxfs/xfs_attr_leaf.c   |  60 +++++++++++---
fs/xfs/libxfs/xfs_attr_leaf.h   |   4 +-
fs/xfs/libxfs/xfs_attr_remote.c |  13 ++--
fs/xfs/libxfs/xfs_bmap.c        |   1 +
fs/xfs/libxfs/xfs_da_btree.c    | 169 +++++++++++++++++++++++++++++++++++++++-
fs/xfs/libxfs/xfs_da_btree.h    |   3 +
fs/xfs/libxfs/xfs_dir2.c        |   5 ++
fs/xfs/libxfs/xfs_dir2.h        |   4 +
fs/xfs/libxfs/xfs_dir2_block.c  |  42 +++++-----
fs/xfs/libxfs/xfs_dir2_data.c   |  18 +++--
fs/xfs/libxfs/xfs_dir2_leaf.c   | 100 ++++++++++++++++++------
fs/xfs/libxfs/xfs_dir2_node.c   |  44 ++++++-----
fs/xfs/libxfs/xfs_dir2_priv.h   |  15 ++--
fs/xfs/libxfs/xfs_exchmaps.c    |   7 +-
fs/xfs/scrub/attr.c             |   1 +
fs/xfs/scrub/dabtree.c          |   8 ++
fs/xfs/scrub/dir.c              |  23 +++---
fs/xfs/scrub/readdir.c          |   6 +-
fs/xfs/xfs_attr_item.c          |   1 +
fs/xfs/xfs_attr_list.c          |  89 ++++++++++++++-------
fs/xfs/xfs_dir2_readdir.c       |   6 +-
fs/xfs/xfs_trace.h              |   7 +-
23 files changed, 492 insertions(+), 148 deletions(-)
#7 lore raw

[GIT PULL 07/16] xfs: online repair of extended attributes

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit fe6c9f8e48e0dcbfc3dba17edd88490c8579b34b:

xfs: validate explicit directory free block owners (2024-04-15 14:58:52 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-xattrs-6.10_2024-04-15

for you to fetch changes up to 6c631e79e73c7122c890ef943f8ca9aab9e1dec8:

xfs: create an xattr iteration function for scrub (2024-04-15 14:58:54 -0700)

----------------------------------------------------------------
xfs: online repair of extended attributes [v30.3 07/16]

This series employs atomic extent swapping to enable safe reconstruction of extended attribute data attached to a file. Because xattrs do not have any redundant information to draw off of, we can at best salvage as much data as we can and build a new structure.

Rebuilding an extended attribute structure consists of these three steps:

First, we walk the existing attributes to salvage as many of them as we can, by adding them as new attributes attached to the repair tempfile. We need to add a new xfile-based data structure to hold blobs of arbitrary length to stage the xattr names and values.

Second, we write the salvaged attributes to a temporary file, and use atomic extent swaps to exchange the entire attribute fork between the two files.

Finally, we reap the old xattr blocks (which are now in the temporary file) as carefully as we can.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (7):
xfs: enable discarding of folios backing an xfile
xfs: create a blob array data structure
xfs: use atomic extent swapping to fix user file fork data
xfs: repair extended attributes
xfs: scrub should set preen if attr leaf has holes
xfs: flag empty xattr leaf blocks for optimization
xfs: create an xattr iteration function for scrub

fs/xfs/Makefile               |    3 +
fs/xfs/libxfs/xfs_attr.c      |    2 +-
fs/xfs/libxfs/xfs_attr.h      |    2 +
fs/xfs/libxfs/xfs_da_format.h |    5 +
fs/xfs/libxfs/xfs_exchmaps.c  |    2 +-
fs/xfs/libxfs/xfs_exchmaps.h  |    1 +
fs/xfs/scrub/attr.c           |  158 +++---
fs/xfs/scrub/attr.h           |    7 +
fs/xfs/scrub/attr_repair.c    | 1207 +++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/attr_repair.h    |   11 +
fs/xfs/scrub/dab_bitmap.h     |   37 ++
fs/xfs/scrub/dabtree.c        |   16 +
fs/xfs/scrub/dabtree.h        |    3 +
fs/xfs/scrub/listxattr.c      |  312 +++++++++++
fs/xfs/scrub/listxattr.h      |   17 +
fs/xfs/scrub/repair.c         |   46 ++
fs/xfs/scrub/repair.h         |    6 +
fs/xfs/scrub/scrub.c          |    2 +-
fs/xfs/scrub/tempexch.h       |    2 +
fs/xfs/scrub/tempfile.c       |  204 +++++++
fs/xfs/scrub/tempfile.h       |    3 +
fs/xfs/scrub/trace.h          |   85 +++
fs/xfs/scrub/xfarray.c        |   17 +
fs/xfs/scrub/xfarray.h        |    2 +
fs/xfs/scrub/xfblob.c         |  168 ++++++
fs/xfs/scrub/xfblob.h         |   26 +
fs/xfs/scrub/xfile.c          |   12 +
fs/xfs/scrub/xfile.h          |    6 +
fs/xfs/xfs_buf.c              |    3 +
fs/xfs/xfs_trace.h            |    2 +
30 files changed, 2284 insertions(+), 83 deletions(-)

create mode 100644 fs/xfs/scrub/attr_repair.c
create mode 100644 fs/xfs/scrub/attr_repair.h
create mode 100644 fs/xfs/scrub/dab_bitmap.h
create mode 100644 fs/xfs/scrub/listxattr.c
create mode 100644 fs/xfs/scrub/listxattr.h
create mode 100644 fs/xfs/scrub/xfblob.c
create mode 100644 fs/xfs/scrub/xfblob.h

#8 lore raw

[GIT PULL 08/16] xfs: online repair of inode unlinked state

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 6c631e79e73c7122c890ef943f8ca9aab9e1dec8:

xfs: create an xattr iteration function for scrub (2024-04-15 14:58:54 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-unlinked-inode-state-6.10_2024-04-15

for you to fetch changes up to 669dfe883c8e20231495f80a28ec7cc0b8fdddc4:

xfs: update the unlinked list when repairing link counts (2024-04-15 14:58:55 -0700)

----------------------------------------------------------------
xfs: online repair of inode unlinked state [v30.3 08/16]

This series adds some logic to the inode scrubbers so that they can detect and deal with consistency errors between the link count and the per-inode unlinked list state. The helpers needed to do this are presented here because they are a prequisite for rebuildng directories, since we need to get a rebuilt non-empty directory off the unlinked list.

Note that this patchset does not provide comprehensive reconstruction of the AGI unlinked list; that is coming in a subsequent patchset.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (2):
xfs: ensure unlinked list state is consistent with nlink during scrub
xfs: update the unlinked list when repairing link counts

fs/xfs/scrub/inode.c         | 19 +++++++++++++++++++
fs/xfs/scrub/inode_repair.c  | 45 ++++++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/nlinks_repair.c | 42 ++++++++++++++++++++++++++++++++---------
fs/xfs/xfs_inode.c           |  5 +----
fs/xfs/xfs_inode.h           |  2 ++
5 files changed, 100 insertions(+), 13 deletions(-)
#9 lore raw

[GIT PULL 09/16] xfs: online repair of directories

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 669dfe883c8e20231495f80a28ec7cc0b8fdddc4:

xfs: update the unlinked list when repairing link counts (2024-04-15 14:58:55 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-dirs-6.10_2024-04-15

for you to fetch changes up to 34c9382c128270d0f4c8b36783b30f3c8085b2dd:

xfs: ask the dentry cache if it knows the parent of a directory (2024-04-15 14:58:56 -0700)

----------------------------------------------------------------
xfs: online repair of directories [v30.3 09/16]

This series employs atomic extent swapping to enable safe reconstruction of directory data. For now, XFS does not support reverse directory links (aka parent pointers), so we can only salvage the dirents of a directory and construct a new structure.

Directory repair therefore consists of five main parts:

First, we walk the existing directory to salvage as many entries as we can, by adding them as new directory entries to the repair temp dir.

Second, we validate the parent pointer found in the directory. If one was not found, we scan the entire filesystem looking for a potential parent.

Third, we use atomic extent swaps to exchange the entire data fork between the two directories.

Fourth, we reap the old directory blocks as carefully as we can.

To wrap up the directory repair code, we need to add to the regular filesystem the ability to free all the data fork blocks in a directory. This does not change anything with normal directories, since they must still unlink and shrink one entry at a time. However, this will facilitate freeing of partially-inactivated temporary directories during log recovery.

The second half of this patchset implements repairs for the dotdot entries of directories. For now there is only rudimentary support for this, because there are no directory parent pointers, so the best we can do is scanning the filesystem and the VFS dcache for answers.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (5):
xfs: inactivate directory data blocks
xfs: online repair of directories
xfs: scan the filesystem to repair a directory dotdot entry
xfs: online repair of parent pointers
xfs: ask the dentry cache if it knows the parent of a directory

fs/xfs/Makefile              |    3 +
fs/xfs/scrub/dir.c           |    9 +
fs/xfs/scrub/dir_repair.c    | 1402 ++++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/findparent.c    |  448 ++++++++++++++
fs/xfs/scrub/findparent.h    |   50 ++
fs/xfs/scrub/inode_repair.c  |    5 +
fs/xfs/scrub/iscan.c         |   18 +-
fs/xfs/scrub/iscan.h         |    1 +
fs/xfs/scrub/nlinks.c        |   23 +
fs/xfs/scrub/nlinks_repair.c |    9 +
fs/xfs/scrub/parent.c        |   14 +-
fs/xfs/scrub/parent_repair.c |  234 +++++++
fs/xfs/scrub/readdir.c       |    7 +
fs/xfs/scrub/repair.c        |    1 +
fs/xfs/scrub/repair.h        |    8 +
fs/xfs/scrub/scrub.c         |    4 +-
fs/xfs/scrub/tempfile.c      |   13 +
fs/xfs/scrub/tempfile.h      |    2 +
fs/xfs/scrub/trace.h         |  115 ++++
fs/xfs/scrub/xfblob.h        |   24 +
fs/xfs/xfs_inode.c           |   51 ++
21 files changed, 2437 insertions(+), 4 deletions(-)

create mode 100644 fs/xfs/scrub/dir_repair.c
create mode 100644 fs/xfs/scrub/findparent.c
create mode 100644 fs/xfs/scrub/findparent.h
create mode 100644 fs/xfs/scrub/parent_repair.c

#10 lore raw

[GIT PULL 10/16] xfs: move orphan files to lost and found

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 34c9382c128270d0f4c8b36783b30f3c8085b2dd:

xfs: ask the dentry cache if it knows the parent of a directory (2024-04-15 14:58:56 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-orphanage-6.10_2024-04-15

for you to fetch changes up to 73597e3e42b4a15030e6f93b71b53a04377ea419:

xfs: ensure dentry consistency when the orphanage adopts a file (2024-04-15 14:58:57 -0700)

----------------------------------------------------------------
xfs: move orphan files to lost and found [v30.3 10/16]

Orphaned files are defined to be files with nonzero ondisk link count but no observable parent directory. This series enables online repair to reparent orphaned files into the filesystem directory tree, and wires up this reparenting ability into the directory, file link count, and parent pointer repair functions. This is how we fix files with positive link count that are not reachable through the directory tree.

This patch will also create the orphanage directory (lost+found) if it is not present. In contrast to xfs_repair, we follow e2fsck in creating the lost+found without group or other-owner access to avoid accidental disclosure of files that were previously hidden by an 0700 directory. That's silly security, but people have been known to do it.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: move orphan files to the orphanage
xfs: move files to orphanage instead of letting nlinks drop to zero
xfs: ensure dentry consistency when the orphanage adopts a file

.../filesystems/xfs/xfs-online-fsck-design.rst     |  20 +-
fs/xfs/Makefile                                    |   1 +
fs/xfs/scrub/dir_repair.c                          | 130 ++++-
fs/xfs/scrub/nlinks.c                              |  20 +-
fs/xfs/scrub/nlinks.h                              |   7 +
fs/xfs/scrub/nlinks_repair.c                       | 123 ++++-
fs/xfs/scrub/orphanage.c                           | 589 +++++++++++++++++++++
fs/xfs/scrub/orphanage.h                           |  75 +++
fs/xfs/scrub/parent_repair.c                       | 100 +++-
fs/xfs/scrub/repair.h                              |   2 +
fs/xfs/scrub/scrub.c                               |   2 +
fs/xfs/scrub/scrub.h                               |   4 +
fs/xfs/scrub/trace.c                               |   1 +
fs/xfs/scrub/trace.h                               |  96 ++++
fs/xfs/xfs_inode.c                                 |   6 +-
fs/xfs/xfs_inode.h                                 |   1 +
16 files changed, 1139 insertions(+), 38 deletions(-)

create mode 100644 fs/xfs/scrub/orphanage.c
create mode 100644 fs/xfs/scrub/orphanage.h

#11 lore raw

[GIT PULL 11/16] xfs: online repair of symbolic links

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 73597e3e42b4a15030e6f93b71b53a04377ea419:

xfs: ensure dentry consistency when the orphanage adopts a file (2024-04-15 14:58:57 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-symlink-6.10_2024-04-15

for you to fetch changes up to 2651923d8d8db00a57665822f017fa7c76758044:

xfs: online repair of symbolic links (2024-04-15 14:58:58 -0700)

----------------------------------------------------------------
xfs: online repair of symbolic links [v30.3 11/16]

The patches in this set adds the ability to repair the target buffer of a symbolic link, using the same salvage, rebuild, and swap strategy used everywhere else.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: expose xfs_bmap_local_to_extents for online repair
xfs: pass the owner to xfs_symlink_write_target
xfs: online repair of symbolic links

fs/xfs/Makefile                    |   1 +
fs/xfs/libxfs/xfs_bmap.c           |  11 +-
fs/xfs/libxfs/xfs_bmap.h           |   6 +
fs/xfs/libxfs/xfs_symlink_remote.c |   7 +-
fs/xfs/libxfs/xfs_symlink_remote.h |   7 +-
fs/xfs/scrub/repair.h              |   8 +
fs/xfs/scrub/scrub.c               |   2 +-
fs/xfs/scrub/symlink.c             |  13 +-
fs/xfs/scrub/symlink_repair.c      | 506 +++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/tempfile.c            |  13 +
fs/xfs/scrub/trace.h               |  46 ++++
fs/xfs/xfs_symlink.c               |   4 +-
12 files changed, 609 insertions(+), 15 deletions(-)

create mode 100644 fs/xfs/scrub/symlink_repair.c

#12 lore raw

[GIT PULL 12/16] xfs: online fsck of iunlink buckets

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 2651923d8d8db00a57665822f017fa7c76758044:

xfs: online repair of symbolic links (2024-04-15 14:58:58 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-iunlink-6.10_2024-04-15

for you to fetch changes up to ab97f4b1c030750f2475bf4da8a9554d02206640:

xfs: repair AGI unlinked inode bucket lists (2024-04-15 14:58:58 -0700)

----------------------------------------------------------------
xfs: online fsck of iunlink buckets [v30.3 12/16]

This series enhances the AGI scrub code to check the unlinked inode bucket lists for errors, and fixes them if necessary. Now that iunlink pointer updates are virtual log items, we can batch updates pretty efficiently in the logging code.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: check AGI unlinked inode buckets
xfs: hoist AGI repair context to a heap object
xfs: repair AGI unlinked inode bucket lists

fs/xfs/scrub/agheader.c        |  40 ++
fs/xfs/scrub/agheader_repair.c | 879 ++++++++++++++++++++++++++++++++++++++---
fs/xfs/scrub/agino_bitmap.h    |  49 +++
fs/xfs/scrub/trace.h           | 255 ++++++++++++
fs/xfs/xfs_inode.c             |   2 +-
fs/xfs/xfs_inode.h             |   1 +
6 files changed, 1179 insertions(+), 47 deletions(-)

create mode 100644 fs/xfs/scrub/agino_bitmap.h

#13 lore raw

[GIT PULL 13/16] xfs: inode-related repair fixes

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit ab97f4b1c030750f2475bf4da8a9554d02206640:

xfs: repair AGI unlinked inode bucket lists (2024-04-15 14:58:58 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/inode-repair-improvements-6.10_2024-04-15

for you to fetch changes up to 1a5f6e08d4e379a23da5be974aee50b26a20c5b0:

xfs: create subordinate scrub contexts for xchk_metadata_inode_subtype (2024-04-15 14:59:00 -0700)

----------------------------------------------------------------
xfs: inode-related repair fixes [v30.3 13/16]

While doing QA of the online fsck code, I made a few observations:
First, nobody was checking that the di_onlink field is actually zero; Second, that allocating a temporary file for repairs can fail (and thus bring down the entire fs) if the inode cluster is corrupt; and Third, that file link counts do not pin at ~0U to prevent integer overflows. Fourth, the x{chk,rep}_metadata_inode_fork functions should be subclassing the main scrub context, not modifying the parent's setup willy-nilly.

This scattered patchset fixes those three problems.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (4):
xfs: check unused nlink fields in the ondisk inode
xfs: try to avoid allocating from sick inode clusters
xfs: pin inodes that would otherwise overflow link count
xfs: create subordinate scrub contexts for xchk_metadata_inode_subtype

fs/xfs/libxfs/xfs_format.h    |  6 ++++
fs/xfs/libxfs/xfs_ialloc.c    | 40 ++++++++++++++++++++++++++
fs/xfs/libxfs/xfs_inode_buf.c |  8 ++++++
fs/xfs/scrub/common.c         | 23 +++------------
fs/xfs/scrub/dir_repair.c     | 11 ++-----
fs/xfs/scrub/inode_repair.c   | 12 ++++++++
fs/xfs/scrub/nlinks.c         |  4 ++-
fs/xfs/scrub/nlinks_repair.c  |  8 ++----
fs/xfs/scrub/repair.c         | 67 +++++++++----------------------------------
fs/xfs/scrub/scrub.c          | 63 ++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/scrub.h          | 11 +++++++
fs/xfs/xfs_inode.c            | 33 ++++++++++++++-------
12 files changed, 187 insertions(+), 99 deletions(-)
#14 lore raw

[GIT PULL 14/16] xfs: less heavy locks during fstrim

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 1a5f6e08d4e379a23da5be974aee50b26a20c5b0:

xfs: create subordinate scrub contexts for xchk_metadata_inode_subtype (2024-04-15 14:59:00 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/discard-relax-locks-6.10_2024-04-15

for you to fetch changes up to b0ffe661fab4b939e4472ef96b8dac3c74e0e03e:

xfs: fix performance problems when fstrimming a subset of a fragmented AG (2024-04-15 14:59:00 -0700)

----------------------------------------------------------------
xfs: less heavy locks during fstrim [v30.3 14/16]

Congratulations! You have made it to the final patchset of the main online fsck feature! This patchset fixes some stalling behavior that I observed when running FITRIM against large flash-based filesystems with very heavily fragmented free space data. In summary -- the current fstrim implementation optimizes for trimming the largest free extents first, and holds the AGF lock for the duration of the operation. This is great if fstrim is being run as a foreground process by a sysadmin.

For xfs_scrub, however, this isn't so good -- we don't really want to block on one huge kernel call while reporting no progress information. We don't want to hold the AGF so long that background processes stall. These problems are easily fixable by issuing smaller FITRIM calls, but there's still the problem of walking the entire cntbt. To solve that second problem, we introduce a new sub-AG FITRIM implementation. To solve the first problem, make it relax the AGF periodically.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (1):
xfs: fix performance problems when fstrimming a subset of a fragmented AG

fs/xfs/xfs_discard.c | 153 +++++++++++++++++++++++++++++++--------------------

1 file changed, 93 insertions(+), 60 deletions(-)
#15 lore raw

[GIT PULL 15/16] xfs: design documentation for online fsck, part 2

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit b0ffe661fab4b939e4472ef96b8dac3c74e0e03e:

xfs: fix performance problems when fstrimming a subset of a fragmented AG (2024-04-15 14:59:00 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/online-fsck-design-6.10_2024-04-15

for you to fetch changes up to 67bdcd499909708195b9408c106b94250955c5ff:

docs: describe xfs directory tree online fsck (2024-04-15 14:59:01 -0700)

----------------------------------------------------------------
xfs: design documentation for online fsck, part 2 [v13.2 15/16]

This series updates the design documentation for online fsck to reflect the final design of the parent pointers feature as well as the implementation of online fsck for the new metadata.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Darrick J. Wong (4):
docs: update the parent pointers documentation to the final version
docs: update online directory and parent pointer repair sections
docs: update offline parent pointer repair strategy
docs: describe xfs directory tree online fsck

.../filesystems/xfs/xfs-online-fsck-design.rst | 354 ++++++++++++++++-----

1 file changed, 266 insertions(+), 88 deletions(-)
#16 lore raw

[GIT PULL 16/16] xfs: retain ILOCK during directory updates

In reply to Darrick J. Wong

Hi Chandan,

Please pull this branch with changes for xfs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems.

--D

The following changes since commit 67bdcd499909708195b9408c106b94250955c5ff:

docs: describe xfs directory tree online fsck (2024-04-15 14:59:01 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/retain-ilock-during-dir-ops-6.10_2024-04-15

for you to fetch changes up to df760471477400ccd3ddcea85d2d6d92f4dad28c:

xfs: unlock new repair tempfiles after creation (2024-04-15 14:59:03 -0700)

----------------------------------------------------------------
xfs: retain ILOCK during directory updates [v13.2 16/16]

This series changes the directory update code to retain the ILOCK on all files involved in a rename until the end of the operation. The upcoming parent pointers patchset applies parent pointers in a separate chained update from the actual directory update, which is why it is now necessary to keep the ILOCK instead of dropping it after the first transaction in the chain.

As a side effect, we no longer need to hold the IOLOCK during an rmapbt scan of inodes to serialize the scan with ongoing directory updates.

This has been running on the djcloud for months with no problems. Enjoy!

----------------------------------------------------------------
Allison Henderson (5):
xfs: Increase XFS_DEFER_OPS_NR_INODES to 5
xfs: Increase XFS_QM_TRANS_MAXDQS to 5
xfs: Hold inode locks in xfs_ialloc
xfs: Hold inode locks in xfs_trans_alloc_dir
xfs: Hold inode locks in xfs_rename

Darrick J. Wong (2):
xfs: don't pick up IOLOCK during rmapbt repair scan
xfs: unlock new repair tempfiles after creation

fs/xfs/libxfs/xfs_defer.c  |  6 ++-
fs/xfs/libxfs/xfs_defer.h  |  8 +++-
fs/xfs/scrub/rmap_repair.c | 16 +-------
fs/xfs/scrub/tempfile.c    |  2 +
fs/xfs/xfs_dquot.c         | 41 +++++++++++++++++++
fs/xfs/xfs_dquot.h         |  1 +
fs/xfs/xfs_inode.c         | 98 ++++++++++++++++++++++++++++++++++------------
fs/xfs/xfs_inode.h         |  2 +
fs/xfs/xfs_qm.c            |  4 +-
fs/xfs/xfs_qm.h            |  2 +-
fs/xfs/xfs_symlink.c       |  6 ++-
fs/xfs/xfs_trans.c         |  9 ++++-
fs/xfs/xfs_trans_dquot.c   | 15 ++++---
13 files changed, 156 insertions(+), 54 deletions(-)