xfs: autonomous self healing
Subsystems (7)
Hi everyone,
You might recall that 18 months ago I showed off an early draft of a patchset implementing autonomous self healing capabilities for XFS. The premise is quite simple -- add a few hooks to the kernel to capture significant filesystem metadata and file health events (pretty much all failures), queue these events to a special anonfd, and let userspace read the events at its leisure. That's patchset 1.
The userspace part is more interesting, because there's a new daemon that opens the anonfd given the root dir of a filesystem, captures a file handle for the root dir, detaches from the root dir, and waits for metadata events. Upon receipt of an adverse health event, it will reopen the root directory and initiate repairs. I've left the prototype Python script in place (patchset 2) but my ultimate goal is for everyone to use the Rust version (patchset 3) because it's much quicker to respond to problems.
New QA tests are patchset 4. Zorro: No need to merge this right away.
This work was mostly complete by the end of 2024, and I've been letting it run on my XFS QA testing fleets ever since then. I am submitting this patchset for upstream for 6.19. Once this is merged, the online fsck project will be complete.
--D
PATCH v2 1/1 xfs: autonomous self healing of filesystems
Hi all,
This patchset builds new functionality to deliver live information about filesystem health events to userspace. This is done by creating an anonymous file that can be read() for events by userspace programs. Events are captured by hooking various parts of XFS and iomap so that metadata health failures, file I/O errors, and major changes in filesystem state (unmounts, shutdowns, etc.) can be observed by programs.
When an event occurs, the hook functions queue an event object to each event anonfd for later processing. Programs must have CAP_SYS_ADMIN to open the anonfd and there's a maximum event lag to prevent resource overconsumption. The events themselves can be read() from the anonfd either as json objects for human readability, or as C structs for daemons.
In userspace, we create a new daemon program that will read the event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some udev rules.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* docs: remove obsolete links in the xfs online repair documentation
* docs: discuss autonomous self healing in the xfs online repair design doc
* xfs: create debugfs uuid aliases
* xfs: create hooks for monitoring health updates
* xfs: create a filesystem shutdown hook
* xfs: create hooks for media errors
* iomap: report buffered read and write io errors to the filesystem
* iomap: report directio read and write errors to callers
* xfs: create file io error hooks
* xfs: create a special file to pass filesystem health to userspace
* xfs: create event queuing, formatting, and discovery infrastructure
* xfs: report metadata health events through healthmon
* xfs: report shutdown events through healthmon
* xfs: report media errors through healthmon
* xfs: report file io errors through healthmon
* xfs: allow reconfiguration of the health monitoring device
* xfs: validate fds against running healthmon
* xfs: add media error reporting ioctl
* xfs: send uevents when major filesystem events happen
--- fs/iomap/internal.h | 2 fs/xfs/libxfs/xfs_fs.h | 173 ++ fs/xfs/libxfs/xfs_health.h | 52 + fs/xfs/xfs_file.h | 36 fs/xfs/xfs_fsops.h | 14 fs/xfs/xfs_healthmon.h | 107 + fs/xfs/xfs_linux.h | 3 fs/xfs/xfs_mount.h | 13 fs/xfs/xfs_notify_failure.h | 44 + fs/xfs/xfs_super.h | 13 fs/xfs/xfs_trace.h | 404 +++++ include/linux/fs.h | 4 include/linux/iomap.h | 2 Documentation/filesystems/vfs.rst | 7 .../filesystems/xfs/xfs-online-fsck-design.rst | 336 +--- fs/iomap/buffered-io.c | 27 fs/iomap/direct-io.c | 4 fs/iomap/ioend.c | 4 fs/xfs/Kconfig | 8 fs/xfs/Makefile | 7 fs/xfs/libxfs/xfs_healthmon.schema.json | 648 +++++++ fs/xfs/xfs_aops.c | 2 fs/xfs/xfs_file.c | 167 ++ fs/xfs/xfs_fsops.c | 75 + fs/xfs/xfs_health.c | 269 +++ fs/xfs/xfs_healthmon.c | 1741 ++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 7 fs/xfs/xfs_notify_failure.c | 135 +- fs/xfs/xfs_super.c | 109 + fs/xfs/xfs_trace.c | 4 lib/seq_buf.c | 1 31 files changed, 4173 insertions(+), 245 deletions(-) create mode 100644 fs/xfs/xfs_healthmon.h create mode 100644 fs/xfs/libxfs/xfs_healthmon.schema.json create mode 100644 fs/xfs/xfs_healthmon.c
PATCH 1/19 docs: remove obsolete links in the xfs online repair documentation
From: Darrick J. Wong <[email protected]>
Online repair is now merged in upstream, no need to point to patchset links anymore.
--- .../filesystems/xfs/xfs-online-fsck-design.rst | 236 +------------------- 1 file changed, 6 insertions(+), 230 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +6 -230
@@ -105,10 +105,8 @@ occur; this capability aids both strategies. TLDR; Show Me the Code! ----------------------- -Code is posted to the kernel.org git trees as follows: -`kernel changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink>`_, -`userspace changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_, and -`QA test changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs>`_. +Kernel and userspace code has been fully merged as of October 2025. + Each kernel patchset adding an online repair function will use the same branch name across the kernel, xfsprogs, and fstests git repos.
@@ -764,12 +762,8 @@ allow the online fsck developers to compare online fsck against offline fsck, and they enable XFS developers to find deficiencies in the code base. Proposed patchsets include -`general fuzzer improvements -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_, `fuzzing baselines -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_, -and `improvements in fuzz testing comprehensiveness -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_. +<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_. Stress Testing --------------
@@ -801,11 +795,6 @@ Success is defined by the ability to run all of these tests without observing any unexpected filesystem shutdowns due to corrupted metadata, kernel hang check warnings, or any other sort of mischief. -Proposed patchsets include `general stress testing -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_ -and the `evolution of existing per-function stress testing -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_. - 4. User Interface =================
@@ -886,10 +875,6 @@ apply as nice of a priority to IO and CPU scheduling as possible. This measure was taken to minimize delays in the rest of the filesystem. No such hardening has been performed for the cron job. -Proposed patchset: -`Enabling the xfs_scrub background service -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_. - Health Reporting ----------------
@@ -912,13 +897,6 @@ notifications and initiate a repair? *Answer*: These questions remain unanswered, but should be a part of the conversation with early adopters and potential downstream users of XFS. -Proposed patchsets include -`wiring up health reports to correction returns -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_ -and -`preservation of sickness info during memory reclaim -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_. - 5. Kernel Algorithms and Data Structures ========================================
@@ -1310,21 +1288,6 @@ Space allocation records are cross-referenced as follows: are there the same number of reverse mapping records for each block as the reference count record claims? -Proposed patchsets are the series to find gaps in -`refcount btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_, -`inode btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and -`rmap btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records; -to find -`mergeable records -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_; -and to -`improve cross referencing with rmap -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_ -before starting a repair. - Checking Extended Attributes ````````````````````````````
@@ -1756,10 +1719,6 @@ For scrub, the drain works as follows: To avoid polling in step 4, the drain provides a waitqueue for scrub threads to be woken up whenever the intent count drops to zero. -The proposed patchset is the -`scrub intent drain series -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_. - .. _jump_labels: Static Keys (aka Jump Label Patching)
@@ -2036,10 +1995,6 @@ The ``xfarray_store_anywhere`` function is used to insert a record in any null record slot in the bag; and the ``xfarray_unset`` function removes a record from the bag. -The proposed patchset is the -`big in-memory array -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_. - Iterating Array Elements ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2172,10 +2127,6 @@ However, it should be noted that these repair functions only use blob storage to cache a small number of entries before adding them to a temporary ondisk file, which is why compaction is not required. -The proposed patchset is at the start of the -`extended attribute repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series. - .. _xfbtree: In-Memory B+Trees
@@ -2214,11 +2165,6 @@ xfiles enables reuse of the entire btree library. Btrees built atop an xfile are collectively known as ``xfbtrees``. The next few sections describe how they actually work. -The proposed patchset is the -`in-memory btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_ -series. - Using xfiles as a Buffer Cache Target ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2459,14 +2405,6 @@ This enables the log to release the old EFI to keep the log moving forwards. EFIs have a role to play during the commit and reaping phases; please see the next section and the section about :ref:`reaping<reaping>` for more details. -Proposed patchsets are the -`bitmap rework -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_ -and the -`preparation for bulk loading btrees -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_. - - Writing the New Tree ````````````````````
@@ -2623,11 +2561,6 @@ The number of records for the inode btree is the number of xfarray records, but the record count for the free inode btree has to be computed as inode chunk records are stored in the xfarray. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding the Space Reference Counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2716,11 +2649,6 @@ Reverse mappings are added to the bag using ``xfarray_store_anywhere`` and removed via ``xfarray_unset``. Bag members are examined through ``xfarray_iter`` loops. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding File Fork Mapping Indices ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2757,11 +2685,6 @@ EXTENTS format instead of BMBT, which may require a conversion. Third, the incore extent map must be reloaded carefully to avoid disturbing any delayed allocation extents. -The proposed patchset is the -`file mapping repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_ -series. - .. _reaping: Reaping Old Metadata Blocks
@@ -2843,11 +2766,6 @@ blocks. As stated earlier, online repair functions use very large transactions to minimize the chances of this occurring. -The proposed patchset is the -`preparation for bulk loading btrees -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_ -series. - Case Study: Reaping After a Regular Btree Repair ````````````````````````````````````````````````
@@ -2943,11 +2861,6 @@ When the walk is complete, the bitmap disunion operation ``(ag_owner_bitmap & btrees. These blocks can then be reaped using the methods outlined above. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - .. _rmap_reap: Case Study: Reaping After Repairing Reverse Mapping Btrees
@@ -2972,11 +2885,6 @@ methods outlined above. The rest of the process of rebuildng the reverse mapping btree is discussed in a separate :ref:`case study<rmap_repair>`. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding the AGFL ```````````````````````````````
@@ -3024,11 +2932,6 @@ more complicated, because computing the correct value requires traversing the forks, or if that fails, leaving the fields invalid and waiting for the fork fsck functions to run. -The proposed patchset is the -`inode -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_ -repair series. - Quota Record Repairs --------------------
@@ -3045,11 +2948,6 @@ checking are obviously bad limits and timer values. Quota usage counters are checked, repaired, and discussed separately in the section about :ref:`live quotacheck <quotacheck>`. -The proposed patchset is the -`quota -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_ -repair series. - .. _fscounters: Freezing to Fix Summary Counters
@@ -3145,11 +3043,6 @@ long enough to check and correct the summary counters. | This bug was fixed in Linux 5.17. | +--------------------------------------------------------------------------+ -The proposed patchset is the -`summary counter cleanup -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_ -series. - Full Filesystem Scans ---------------------
@@ -3277,15 +3170,6 @@ Second, if the incore inode is stuck in some intermediate state, the scan coordinator must release the AGI and push the main filesystem to get the inode back into a loadable state. -The proposed patches are the -`inode scanner -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ -series. -The first user of the new functionality is the -`online quotacheck -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ -series. - Inode Management ````````````````
@@ -3381,12 +3265,6 @@ To capture these nuances, the online fsck code has a separate ``xchk_irele`` function to set or clear the ``DONTCACHE`` flag to get the required release behavior. -Proposed patchsets include fixing -`scrub iget usage -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_ and -`dir iget usage -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_. - .. _ilocking: Locking Inodes
@@ -3443,11 +3321,6 @@ If the dotdot entry changes while the directory is unlocked, then a move or rename operation must have changed the child's parentage, and the scan can exit early. -The proposed patchset is the -`directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ -series. - .. _fshooks: Filesystem Hooks
@@ -3594,11 +3467,6 @@ The inode scan APIs are pretty simple: - ``xchk_iscan_teardown`` to finish the scan -This functionality is also a part of the -`inode scanner -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ -series. - .. _quotacheck: Case Study: Quota Counter Checking
@@ -3686,11 +3554,6 @@ needing to hold any locks for a long duration. If repairs are desired, the real and shadow dquots are locked and their resource counts are set to the values in the shadow dquot. -The proposed patchset is the -`online quotacheck -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ -series. - .. _nlinks: Case Study: File Link Count Checking
@@ -3744,11 +3607,6 @@ shadow information. If no parents are found, the file must be :ref:`reparented <orphanage>` to the orphanage to prevent the file from being lost forever. -The proposed patchset is the -`file link count repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_ -series. - .. _rmap_repair: Case Study: Rebuilding Reverse Mapping Records
@@ -3828,11 +3686,6 @@ scan for reverse mapping records. 12. Free the xfbtree now that it not needed. -The proposed patchset is the -`rmap repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_ -series. - Staging Repairs with Temporary Files on Disk --------------------------------------------
@@ -3971,11 +3824,6 @@ Once a good copy of a data file has been constructed in a temporary file, it must be conveyed to the file being repaired, which is the topic of the next section. -The proposed patches are in the -`repair temporary files -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_ -series. - Logged File Content Exchanges -----------------------------
@@ -4025,11 +3873,6 @@ The new ``XFS_SB_FEAT_INCOMPAT_EXCHRANGE`` incompatible feature flag in the superblock protects these new log item records from being replayed on old kernels. -The proposed patchset is the -`file contents exchange -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_ -series. - +--------------------------------------------------------------------------+ | **Sidebar: Using Log-Incompatible Feature Flags** | +--------------------------------------------------------------------------+
@@ -4323,11 +4166,6 @@ To repair the summary file, write the xfile contents into the temporary file and use atomic mapping exchange to commit the new contents. The temporary file is then reaped. -The proposed patchset is the -`realtime summary repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_ -series. - Case Study: Salvaging Extended Attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4369,11 +4207,6 @@ Salvaging extended attributes is done as follows: 4. Reap the temporary file. -The proposed patchset is the -`extended attribute repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ -series. - Fixing Directories ------------------
@@ -4448,11 +4281,6 @@ Unfortunately, the current dentry cache design doesn't provide a means to walk every child dentry of a specific directory, which makes this a hard problem. There is no known solution. -The proposed patchset is the -`directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ -series. - Parent Pointers ```````````````
@@ -4612,11 +4440,6 @@ a :ref:`directory entry live update hook <liveupdate>` as follows: 7. Reap the temporary directory. -The proposed patchset is the -`parent pointers directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ -series. - Case Study: Repairing Parent Pointers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4662,11 +4485,6 @@ directory reconstruction: 8. Reap the temporary file. -The proposed patchset is the -`parent pointers repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ -series. - Digression: Offline Checking of Parent Pointers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4755,11 +4573,6 @@ connectivity checks: 4. Move on to examining link counts, as we do today. -The proposed patchset is the -`offline parent pointers repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_ -series. - Rebuilding directories from parent pointers in offline repair would be very challenging because xfs_repair currently uses two single-pass scans of the filesystem during phases 3 and 4 to decide which files are corrupt enough to be
@@ -4903,12 +4716,6 @@ Repairing the directory tree works as follows: 6. If the subdirectory has zero paths, attach it to the lost and found. -The proposed patches are in the -`directory tree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_ -series. - - .. _orphanage: The Orphanage
@@ -4973,11 +4780,6 @@ Orphaned files are adopted by the orphanage as follows: 7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all resources. -The proposed patches are in the -`orphanage adoption -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_ -series. - 6. Userspace Algorithms and Data Structures ===========================================
@@ -5091,14 +4893,6 @@ first workqueue's workers until the backlog eases. This doesn't completely solve the balancing problem, but reduces it enough to move on to more pressing issues. -The proposed patchsets are the scrub -`performance tweaks -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_ -and the -`inode scan rebalance -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_ -series. - .. _scrubrepair: Scheduling Repairs
@@ -5179,20 +4973,6 @@ immediately. Corrupt file data blocks reported by phase 6 cannot be recovered by the filesystem. -The proposed patchsets are the -`repair warning improvements -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_, -refactoring of the -`repair data dependency -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_ -and -`object tracking -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_, -and the -`repair scheduling -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_ -improvement series. - Checking Names for Confusable Unicode Sequences -----------------------------------------------
@@ -5372,6 +5152,8 @@ The extra flexibility enables several new use cases: This emulates an atomic device write in software, and can support arbitrary scattered writes. +(This functionality was merged into mainline as of 2025) + Vectorized Scrub ----------------
@@ -5393,13 +5175,7 @@ It is hoped that ``io_uring`` will pick up enough of this functionality that online fsck can use that instead of adding a separate vectored scrub system call to XFS. -The relevant patchsets are the -`kernel vectorized scrub -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_ -and -`userspace vectorized scrub -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_ -series. +(This functionality was merged into mainline as of 2025) Quality of Service Targets for Scrub ------------------------------------
Re: [PATCH 01/19] docs: remove obsolete links in the xfs online repair documentation
Re: [PATCH 01/19] docs: remove obsolete links in the xfs online repair documentation
Quoted 5 lines · Christoph Hellwig
Looks good:
Reviewed-by Christoph Hellwig <[email protected]>Maybe expedite this for 6.18-rc?
Ok. I guess removing obsolete links is a bug fix :)
--D
PATCH 2/19 docs: discuss autonomous self healing in the xfs online repair design doc
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Update the XFS online repair document to describe the motivation and design of the autonomous filesystem healing agent known as xfs_healer.
--- .../filesystems/xfs/xfs-online-fsck-design.rst | 102 ++++++++++++++++++++ 1 file changed, 100 insertions(+), 2 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +100 -2
@@ -166,9 +166,12 @@ The current XFS tools leave several problems unsolved: malicious actors **exploit quirks of Unicode** to place misleading names in directories. +8. **Site Reliability and Support Engineers** would like to reduce the + frequency of incidents requiring **manual intervention**. + Given this definition of the problems to be solved and the actors who would benefit, the proposed solution is a third fsck tool that acts on a running -filesystem. +filesystem, and an autononmous agent that fixes problems as they arise. This new third program has three components: an in-kernel facility to check metadata, an in-kernel facility to repair metadata, and a userspace driver
@@ -203,6 +206,13 @@ Even if a piece of filesystem metadata can only be regenerated by scanning the entire system, the scan can still be done in the background while other file operations continue. +The autonomous self healing agent should listen for metadata health impact +reports coming from the kernel and automatically schedule repairs for the +damaged metadata. +If the required repairs are larger in scope than a single metadata structure, +``xfs_scrub`` should be invoked to perform a full analysis. +``xfs_healer`` is the name of this program. + In summary, online fsck takes advantage of resource sharding and redundant metadata to enable targeted checking and repair operations while the system is running.
@@ -850,11 +860,16 @@ variable in the following service files: * ``xfs_scrub_all_fail.service`` The decision to enable the background scan is left to the system administrator. -This can be done by enabling either of the following services: +This can be done system-wide by enabling either of the following services: * ``xfs_scrub_all.timer`` on systemd systems * ``xfs_scrub_all.cron`` on non-systemd systems +To enable online repair for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To enable only scanning, the property should be set to ``check``. +To disable online fsck entirely, the property should be set to ``none``. + This automatic weekly scan is configured out of the box to perform an additional media scan of all file data once per month. This is less foolproof than, say, storing file data block checksums, but much
@@ -897,6 +912,36 @@ notifications and initiate a repair? *Answer*: These questions remain unanswered, but should be a part of the conversation with early adopters and potential downstream users of XFS. +Autonomous Self Healing +----------------------- + +The autonomous self healing agent is a background system service that starts +when the filesystem is mounted and runs until unmount. +When starting up, the agent opens a special pseudofile under the specific +mount. +When the filesystem generates new adverse health events, the events will be +made available for reading via the special pseudofile. +The events need not be limited to metadata concerns; they can also reflect +events outside of the filesystem's direct control such as file I/O errors. + +The agent reads these events in a loop and responds to the events +appropriately. +For a single trouble report about metadata, the agent initiates a targeted +repair of the specific structure. +If that repair fails or the agent observes too many metadata trouble reports +over a short interval, it should then initiate a full scan of the filesystem +via the ``xfs_scrub`` service. + +The decision to enable the background scan is left to the system administrator. +This can be done system-wide by enabling the following services: + +* ``[email protected]`` on systemd systems + +To enable autonomous healing for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To disable self healing, the property should be set to ``check``, +``optimize``, or ``none``. + 5. Kernel Algorithms and Data Structures ========================================
@@ -5071,6 +5116,59 @@ and report what has been lost. For media errors in blocks owned by files, parent pointers can be used to construct file paths from inode numbers for user-friendly reporting. +Autonomous Self Healing +----------------------- + +When a filesystem mounts, the Linux kernel initiates a uevent describing the +mount and the path to the data device. +A udev rule determines the initial mountpoint from the data device path +and starts a mount-specific ``xfs_healer`` service instance. +The ``xfs_healer`` service opens the mountpoint and issues the +XFS_IOC_HEALTH_MONITOR ioctl to open a special health monitoring file. +After that is set up, the mountpoint is closed to avoid pinning the mount. + +The health monitoring file hooks certain points of the filesystem so that it +may receive events about metadata health, filesystem shutdowns, media errors, +file I/O errors, and unmounting of the filesystem. +Events are queued up for each health monitor file and encoded into a +``struct xfs_health_monitor_event`` object when the agent calls ``read()`` on +the file. +All health events are dispatched to a background threadpool to reduce stalls +in the main event loop. +Events can be logged into the system log for further analysis. + +For metadata health events, the specific details are used to construct a call +to the scrub ioctl. +The filesystem mountpoint is reopened, and the kernel is called. +If events are lost or the repairs fail, a full scan will be initiated by +starting up an ``[email protected]`` for the given mountpoint. + +A filesystem shutdown causes all future repair work to cease, and an unmount +causes the agent to exit. + +**Question**: Why use a pseudofile and not use existing notification methods? + +*Answer*: The pseudofile is a private filesystem interface only available to +processes with the CAP_SYS_ADMIN priviledge. +Being private gives the kernel and ``xfs_healer`` the flexibility to change +or update the event format in the future without worrying about backwards +compatibility. +Using existing notifications means that the event format would be frozen in +public UAPI forever. + +The pseudofile can also accept ioctls, which gives ``xfs_healer`` a solid +means to validate that prior to a repair, its reopened mountpoint is actually +the same filesystem that is being monitored. + +**Future Work Question**: Should the healer daemon also register a dbus +listener and publish events there? + +*Answer*: This is unclear -- if there's a demand for system monitoring daemons +to consume this information and make decisions, then yes, this could be wired +up in ``xfs_healer``. +On the other hand, systemd is in the middle of a transition to varlink, so +it makes more sense to wait and see what happens. + 7. Conclusion and Future Work =============================
Re: [PATCH 02/19] docs: discuss autonomous self healing in the xfs online repair design doc
Quoted 6 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Update the XFS online repair document to describe the motivation and design of the autonomous filesystem healing agent known as
xfs_healer.Signed-off-by "Darrick J. Wong" <[email protected]>
/me decides (or rather it was pointed out to me) that there's a kernel component to xfs_healer, but no explicit discussion of it in section 5 ("Kernel Algorithms and Data Structures"). Also given the frequency of the question "why not reuse fsnotify?" I'll address the reasons for that here.
I've added the following text, which will appear in the next revision:
5. Kernel Algorithms and Data Structures
========================================
<snip>
+Health Monitoring
+-----------------
+
+A self-correcting filesystem responds to observations of problems by scheduling +repairs of the affected areas.
+The filesystem must therefore create event objects in response to stimuli +(metadata corruption, file I/O errors, etc.) and dispatch these events to +downstream consumers.
+Downstream consumers that are in the kernel itself are easy to implement with +the ``xfs_hooks`` infrastructure created for other parts of online repair; these +are basically indirect function calls.
+
+However, the decision to translate an adverse metadata health report into a +repair should be made by userspace, and the actual scheduling done by userspace. +Some users (e.g. containers) would prefer to fast-fail the container and restart +it on another node at a previous checkpoint.
+For workloads running in isolation, repairs may be preferable; either way this +is something the system administrator knows, and not the kernel.
+A userspace agent (``xfs_healer``, described later) will collect events from the +kernel and dispatch them appropriately.
+
+Exporting health events to userspace requires the creation of a new component, +known as the health monitor.
+Because the monitor exposes itself to userspace to deliver information, a file +descriptor is the natural abstraction to use here.
+The health monitor hooks all the relevant sources of metadata health events. +Upon activation of the hook, a new event object is created and added to a queue. +When the agent reads from the fd, event objects are pulled from the start of the +queue and formatted into the user's buffer.
+The events are freed, and the read call returns to userspace to allow the agent +to perform some work.
+Memory usage is constrained on a per-fd basis to prevent memory exhaustion; if +an event must be discarded, a special "lost event" event is delivered to the +agent.
+
+In short, health events are captured, queued, and eventually copied out to +userspace for dispatching.
+
+**Question**: Why use a pseudofile and not use existing notification methods? +
+*Answer*: The pseudofile is a private filesystem interface only available to +processes with the CAP_SYS_ADMIN priviledge and the ability to open the root +directory.
+Being private gives the kernel and ``xfs_healer`` the flexibility to change +or update the event format in the future without worrying about backwards +compatibility.
+Using existing notifications means that the event format would be frozen in +the public fsnotify UAPI forever, which would affect two subsystems. +
+The pseudofile can also accept ioctls, which gives ``xfs_healer`` a solid +means to validate that prior to a repair, its reopened mountpoint is actually +the same filesystem that is being monitored.
+
+**Question**: Why not reuse fs/notify?
+
+*Answer*: It's much simpler for the healthmon code to manage its own queue of +events and to wake up readers instead of reusing fsnotify because that's the +only part of fsnotify that would use.
+
+Before I get started, an introduction: fsnotify expects its users (e.g. +fanotify) to implement quite a bit of functionality; all it provides is a +wrapper around a simple queue and a lot of code to convey information about the +calling process to that user.
+fanotify has to actually implement all the queue management code on its own, +and so would healthmon.
+
+So if healthmon used fsnotify, it would have to create its own fsnotify group +structure.
+For our purposes, the group is a very large wrapper around a linked list, some +counters, and a mutex.
+The group object is critical for ensuring that sees only its own events, and +that nobody else (e.g. regular fanotify) ever sees these events. +There's a lot more in there for controlling whether fanotify reports pids, +groups, file handles, etc. that healthmon doesn't care about.
+
+Starting from the fsnotify() function call:
+
+ - I /think/ we'd have to define a new "data type", which itself is just a plain
+ int but I think they correspond to FSNOTIFY_EVENT_* values which themselves
+ are actually part of an enum.
+ The data type controls the typecasting options for the ``void *data``
+ parameter, which I guess is how I'd pass the healthmon event info from the
+ hooks into the fsnotify mechanism and back out to the healthmon code.
+
+ - Each filesystem that wants to do this probably has to add their own
+ FSNOTIFY_EVENT_{XFS,BTRFS,BFS} data type value because that's a casting
+ decision that's made inside the main fsnotify code.
+ I think this can be avoided if each fs is careful never to leak events
+ outside of the group.
+ Either way, it's harder to follow the data flows here because fsnotify can
+ only take and pass around ``void *`` pointers, and it makes various indirect
+ function calls to manage events.
+ Contrast this with doing everything with typed pointers and direct calls
+ within ``xfs_healthmon.c``.
+
+ - Since healthmon is both producer and consumer of fsnotify events, we can
+ probably define our own "mask" value.
+ It's a relief that we don't have to interact with fanotify, because fanotify
+ has used up 22 of its 32 mask bits.+
+Once healthmon gets an event into fsnotify, fsnotify will call back (into +healthmon!) to tell it that it got an event.
+From there, the fsnotify implementation (healthmon) has to allocate an event +object and add it to the event queue in the group, which is what it already does +now.
+Overflow control is up to the fsnotify implementation, which healthmon already +implements.
+
+After the event is queued, the fsnotify implementation also has to implement its +own read file op to dequeue an event and copy it to the userspace buffer in +whatever format it likes.
+Again, healthmon already does all this.
+
+In the end, replacing the homegrown event dispatching in healthmon with fsnotify +would make the data flows much harder to understand, and all we gain is a +generic event dispatcher that relies on indirect function calls instead of +direct ones.
+We still have to implement the queuing discipline ourselves! :(
+
+**Future Work Question**: Should these events be exposed through the fanotify +filesystem error event interface?
+
+*Answer*: Yes.
+fanotify is much more careful about filtering out events to processes that +aren't running with privileges.
+These processes should have a means to receive simple notifications about +file errors.
+However, this will require coordination between fanotify, ext4, and XFS, and +is (for now) outside the scope of this project.
--D
Quoted 154 lines
--- .../filesystems/xfs/xfs-online-fsck-design.rst | 102 ++++++++++++++++++++ 1 file changed, 100 insertions(+), 2 deletions(-)Documentation/filesystems/xfs/xfs-online-fsck-design.rst +100
-2@@ -166,9 +166,12 @@ The current XFS tools leave several problems unsolved: malicious actors **exploit quirks of Unicode** to place misleading names in directories. +8. **Site Reliability and Support Engineers** would like to reduce the + frequency of incidents requiring **manual intervention**. + Given this definition of the problems to be solved and the actors who would benefit, the proposed solution is a third fsck tool that acts on a running -filesystem. +filesystem, and an autononmous agent that fixes problems as they arise. This new third program has three components: an in-kernel facility to check metadata, an in-kernel facility to repair metadata, and a userspace driver@@ -203,6 +206,13 @@ Even if a piece of filesystem metadata can only be regenerated by scanning the entire system, the scan can still be done in the background while other file operations continue. +The autonomous self healing agent should listen for metadata health impact +reports coming from the kernel and automatically schedule repairs for the +damaged metadata. +If the required repairs are larger in scope than a single metadata structure, +``xfs_scrub`` should be invoked to perform a full analysis. +``xfs_healer`` is the name of this program. + In summary, online fsck takes advantage of resource sharding and redundant metadata to enable targeted checking and repair operations while the system is running.@@ -850,11 +860,16 @@ variable in the following service files: * ``xfs_scrub_all_fail.service`` The decision to enable the background scan is left to the system administrator. -This can be done by enabling either of the following services: +This can be done system-wide by enabling either of the following services: * ``xfs_scrub_all.timer`` on systemd systems * ``xfs_scrub_all.cron`` on non-systemd systems +To enable online repair for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To enable only scanning, the property should be set to ``check``. +To disable online fsck entirely, the property should be set to ``none``. + This automatic weekly scan is configured out of the box to perform an additional media scan of all file data once per month. This is less foolproof than, say, storing file data block checksums, but much@@ -897,6 +912,36 @@ notifications and initiate a repair? *Answer*: These questions remain unanswered, but should be a part of the conversation with early adopters and potential downstream users of XFS. +Autonomous Self Healing +----------------------- + +The autonomous self healing agent is a background system service that starts +when the filesystem is mounted and runs until unmount. +When starting up, the agent opens a special pseudofile under the specific +mount. +When the filesystem generates new adverse health events, the events will be +made available for reading via the special pseudofile. +The events need not be limited to metadata concerns; they can also reflect +events outside of the filesystem's direct control such as file I/O errors. + +The agent reads these events in a loop and responds to the events +appropriately. +For a single trouble report about metadata, the agent initiates a targeted +repair of the specific structure. +If that repair fails or the agent observes too many metadata trouble reports +over a short interval, it should then initiate a full scan of the filesystem +via the ``xfs_scrub`` service. + +The decision to enable the background scan is left to the system administrator. +This can be done system-wide by enabling the following services: + +* ``[email protected]`` on systemd systems + +To enable autonomous healing for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To disable self healing, the property should be set to ``check``, +``optimize``, or ``none``. + 5. Kernel Algorithms and Data Structures ========================================@@ -5071,6 +5116,59 @@ and report what has been lost. For media errors in blocks owned by files, parent pointers can be used to construct file paths from inode numbers for user-friendly reporting. +Autonomous Self Healing +----------------------- + +When a filesystem mounts, the Linux kernel initiates a uevent describing the +mount and the path to the data device. +A udev rule determines the initial mountpoint from the data device path +and starts a mount-specific ``xfs_healer`` service instance. +The ``xfs_healer`` service opens the mountpoint and issues the +XFS_IOC_HEALTH_MONITOR ioctl to open a special health monitoring file. +After that is set up, the mountpoint is closed to avoid pinning the mount. + +The health monitoring file hooks certain points of the filesystem so that it +may receive events about metadata health, filesystem shutdowns, media errors, +file I/O errors, and unmounting of the filesystem. +Events are queued up for each health monitor file and encoded into a +``struct xfs_health_monitor_event`` object when the agent calls ``read()`` on +the file. +All health events are dispatched to a background threadpool to reduce stalls +in the main event loop. +Events can be logged into the system log for further analysis. + +For metadata health events, the specific details are used to construct a call +to the scrub ioctl. +The filesystem mountpoint is reopened, and the kernel is called. +If events are lost or the repairs fail, a full scan will be initiated by +starting up an ``[email protected]`` for the given mountpoint. + +A filesystem shutdown causes all future repair work to cease, and an unmount +causes the agent to exit. + +**Question**: Why use a pseudofile and not use existing notification methods? + +*Answer*: The pseudofile is a private filesystem interface only available to +processes with the CAP_SYS_ADMIN priviledge. +Being private gives the kernel and ``xfs_healer`` the flexibility to change +or update the event format in the future without worrying about backwards +compatibility. +Using existing notifications means that the event format would be frozen in +public UAPI forever. + +The pseudofile can also accept ioctls, which gives ``xfs_healer`` a solid +means to validate that prior to a repair, its reopened mountpoint is actually +the same filesystem that is being monitored. + +**Future Work Question**: Should the healer daemon also register a dbus +listener and publish events there? + +*Answer*: This is unclear -- if there's a demand for system monitoring daemons +to consume this information and make decisions, then yes, this could be wired +up in ``xfs_healer``. +On the other hand, systemd is in the middle of a transition to varlink, so +it makes more sense to wait and see what happens. + 7. Conclusion and Future Work =============================
PATCH 3/19 xfs: create debugfs uuid aliases
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an alias for the debugfs dir so that we can find a filesystem by uuid. Unless it's mounted nouuid.
--- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
fs/xfs/xfs_mount.h +1 -0
@@ -290,6 +290,7 @@ typedef struct xfs_mount {struct delayed_work m_reclaim_work;/* background inode reclaim */ struct xfs_zone_info *m_zone_info;/* zone allocator information */ struct dentry *m_debugfs;/* debugfs parent */ +struct dentry *m_debugfs_uuid;/* debugfs symlink */ struct xfs_kobj m_kobj;struct xfs_kobj m_error_kobj;struct xfs_kobj m_error_meta_kobj;
fs/xfs/xfs_super.c +11 -0
@@ -813,6 +813,7 @@ xfs_mount_free(if (mp->m_ddev_targp)xfs_free_buftarg (mp->m_ddev_targp); +debugfs_remove (mp->m_debugfs_uuid);debugfs_remove (mp->m_debugfs);kfree (mp->m_rtname);kfree (mp->m_logname);
@@ -1963,6 +1964,16 @@ xfs_fs_fill_super( goto out_unmount; } +if (xfs_debugfs && mp->m_debugfs && !xfs_has_nouuid (mp)) { +char name[UUID_STRING_LEN +1 ]; + +snprintf (name,UUID_STRING_LEN +1 ,"%pU" , &mp->m_sb.sb_uuid); + mp->m_debugfs_uuid =debugfs_create_symlink (name, xfs_debugfs, + mp->m_super->s_id); + }else { + mp->m_debugfs_uuid =NULL ; + } +return 0 ; out_filestream_unmount:
PATCH 4/19 xfs: create hooks for monitoring health updates
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create hooks for monitoring health events.
--- fs/xfs/libxfs/xfs_health.h | 47 ++++++++++ fs/xfs/xfs_mount.h | 3 + fs/xfs/xfs_health.c | 202 ++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_super.c | 1 4 files changed, 252 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_health.h +47 -0
@@ -289,4 +289,51 @@ void xfs_bulkstat_health(struct xfs_inode *ip, struct xfs_bulkstat *bs);#define xfs_metadata_is_sick (error) \ (unlikely((error) == -EFSCORRUPTED || (error) == -EFSBADCRC)) +/* +* Parameters for tracking health updates. The enum below is passed as the +* hook function argument. +*/ +enum xfs_health_update_type { +XFS_HEALTHUP_SICK =1 ,/* runtime corruption observed */ +XFS_HEALTHUP_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHUP_HEALTHY ,/* fsck reported healthy structure */ +XFS_HEALTHUP_UNMOUNT ,/* filesystem is unmounting */ +}; + +/* Where in the filesystem was the event observed? */ +enum xfs_health_update_domain { +XFS_HEALTHUP_FS =1 ,/* main filesystem */ +XFS_HEALTHUP_AG ,/* allocation group */ +XFS_HEALTHUP_INODE ,/* inode */ +XFS_HEALTHUP_RTGROUP ,/* realtime group */ +}; + +struct xfs_health_update_params { +/* XFS_HEALTHUP_INODE */ +xfs_ino_t ino; +uint32_t gen; + +/* XFS_HEALTHUP_AG/RTGROUP */ +uint32_t group; + +/* XFS_SICK_* flags */ +unsigned int old_mask; +unsigned int new_mask; + +enum xfs_health_update_domain domain; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_health_hook { +struct xfs_hook health_hook; +}; + +void xfs_health_hook_disable (void ); +void xfs_health_hook_enable (void ); + +int xfs_health_hook_add (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_del (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_HEALTH_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -344,6 +344,9 @@ typedef struct xfs_mount {/* Hook to feed dirent updates to an active online repair. */ struct xfs_hooks m_dir_update_hooks; + +/* Hook to feed health events to a daemon. */ +struct xfs_hooks m_health_update_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_health.c +201 -1
348 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2285,6 +2285,7 @@ xfs_init_fs_context( mp->m_allocsize_log =16 ;/* 64k */ xfs_hooks_init (&mp->m_dir_update_hooks); +xfs_hooks_init (&mp->m_health_update_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 5/19 xfs: create a filesystem shutdown hook
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a hook so that health monitoring can report filesystem shutdown events to userspace.
--- fs/xfs/xfs_fsops.h | 14 +++++++++++++ fs/xfs/xfs_mount.h | 3 +++ fs/xfs/xfs_fsops.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_super.c | 1 + 4 files changed, 75 insertions(+)
fs/xfs/xfs_fsops.h +14 -0
@@ -15,4 +15,18 @@ int xfs_fs_goingdown(struct xfs_mount *mp, uint32_t inflags);int xfs_fs_reserve_ag_blocks (struct xfs_mount *mp);void xfs_fs_unreserve_ag_blocks (struct xfs_mount *mp); +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_shutdown_hook { +struct xfs_hook shutdown_hook; +}; + +void xfs_shutdown_hook_disable (void ); +void xfs_shutdown_hook_enable (void ); + +int xfs_shutdown_hook_add (struct xfs_mount *mp,struct xfs_shutdown_hook *hook); +void xfs_shutdown_hook_del (struct xfs_mount *mp,struct xfs_shutdown_hook *hook); +void xfs_shutdown_hook_setup (struct xfs_shutdown_hook *hook, +notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_FSOPS_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -347,6 +347,9 @@ typedef struct xfs_mount {/* Hook to feed health events to a daemon. */ struct xfs_hooks m_health_update_hooks; + +/* Hook to feed shutdown events to a daemon. */ +struct xfs_hooks m_shutdown_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_fsops.c +57 -0
69 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2285,6 +2285,7 @@ xfs_init_fs_context( mp->m_allocsize_log =16 ;/* 64k */ xfs_hooks_init (&mp->m_dir_update_hooks); +xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks); fc->s_fs_info = mp;
PATCH 6/19 xfs: create hooks for media errors
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a media error event hook so that we can send events to userspace.
--- fs/xfs/xfs_mount.h | 3 ++ fs/xfs/xfs_notify_failure.h | 38 +++++++++++++++++++ fs/xfs/xfs_notify_failure.c | 84 ++++++++++++++++++++++++++++++++++++++++--- fs/xfs/xfs_super.c | 1 + 4 files changed, 121 insertions(+), 5 deletions(-)
fs/xfs/xfs_mount.h +3 -0
@@ -350,6 +350,9 @@ typedef struct xfs_mount {/* Hook to feed shutdown events to a daemon. */ struct xfs_hooks m_shutdown_hooks; + +/* Hook to feed media error events to a daemon. */ +struct xfs_hooks m_media_error_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_notify_failure.h +38 -0
@@ -8,4 +8,42 @@extern const struct dax_holder_operations xfs_dax_holder_operations; +enum xfs_failed_device { +XFS_FAILED_DATADEV , +XFS_FAILED_LOGDEV , +XFS_FAILED_RTDEV , +}; + +#if defined(CONFIG_XFS_LIVE_HOOKS ) && defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX ) +struct xfs_media_error_params { +struct xfs_mount *mp; +enum xfs_failed_device fdev; +xfs_daddr_t daddr; +uint64_t bbcount; +bool pre_remove; +}; + +struct xfs_media_error_hook { +struct xfs_hook error_hook; +}; + +void xfs_media_error_hook_disable (void ); +void xfs_media_error_hook_enable (void ); + +int xfs_media_error_hook_add (struct xfs_mount *mp, +struct xfs_media_error_hook *hook); +void xfs_media_error_hook_del (struct xfs_mount *mp, +struct xfs_media_error_hook *hook); +void xfs_media_error_hook_setup (struct xfs_media_error_hook *hook, +notifier_fn_t mod_fn); +#else +struct xfs_media_error_params { }; +struct xfs_media_error_hook { }; +# define xfs_media_error_hook_disable () ((void)0) +# define xfs_media_error_hook_enable () ((void)0) +# define xfs_media_error_hook_add (...) (0) +# define xfs_media_error_hook_del (...) ((void)0) +# define xfs_media_error_hook_setup (...) ((void)0) +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_notify_failure.c +79 -5
107 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2287,6 +2287,7 @@ xfs_init_fs_context(xfs_hooks_init (&mp->m_dir_update_hooks);xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks); +xfs_hooks_init (&mp->m_media_error_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 7/19 iomap: report buffered read and write io errors to the filesystem
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Provide a callback so that iomap can report read and write IO errors to the caller filesystem. For now this is only wired up for iomap as a testbed for XFS.
--- fs/iomap/internal.h | 2 ++ include/linux/fs.h | 4 ++++ Documentation/filesystems/vfs.rst | 7 +++++++ fs/iomap/buffered-io.c | 27 +++++++++++++++++++++++++-- fs/iomap/ioend.c | 4 ++++ 5 files changed, 42 insertions(+), 2 deletions(-)
fs/iomap/internal.h +2 -0
@@ -5,5 +5,7 @@#define IOEND_BATCH_SIZE 4096u32 iomap_finish_ioend_direct (struct iomap_ioend *ioend); +void iomap_mapping_ioerror (struct address_space *mapping,int direction, +loff_t pos,u64 len,int error);#endif /* _IOMAP_INTERNAL_H */
include/linux/fs.h +4 -0
@@ -477,6 +477,10 @@ struct address_space_operations { sector_t *span);void (*swap_deactivate)(struct file *file);int (*swap_rw)(struct kiocb *iocb,struct iov_iter *iter); + +/* Callback for dealing with IO errors during readahead or writeback */ +void (*ioerror)(struct address_space *mapping,int direction, +loff_t pos,u64 len,int error); };extern const struct address_space_operations empty_aops;
Documentation/filesystems/vfs.rst +7 -0
@@ -822,6 +822,8 @@ cache in your filesystem. The following members are defined: int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); + void (*ioerror)(struct address_space *mapping, int direction, + loff_t pos, u64 len, int error); }; ``read_folio``
@@ -1032,6 +1034,11 @@ cache in your filesystem. The following members are defined: ``swap_rw`` Called to read or write swap pages when SWP_FS_OPS is set. +``ioerror`` + Called to deal with IO errors during readahead or writeback. + This may be called from interrupt context, and without any + locks necessarily being held. + The File Object ===============
fs/iomap/buffered-io.c +25 -2
@@ -288,6 +288,14 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter, pos >=i_size_read (iter->inode); } +inline void iomap_mapping_ioerror (struct address_space *mapping,int direction, +loff_t pos,u64 len,int error) +{ +if (mapping && mapping->a_ops->ioerror) + mapping->a_ops->ioerror (mapping, direction, pos, len, + error); +} +/** * iomap_read_inline_data - copy inline data into the page cache * @iter: iteration structure
@@ -310,8 +318,11 @@ static int iomap_read_inline_data(const struct iomap_iter *iter,if (folio_test_uptodate (folio))return 0 ; -if (WARN_ON_ONCE (size > iomap->length)) +if (WARN_ON_ONCE (size > iomap->length)) { +iomap_mapping_ioerror (folio->mapping,READ , iomap->offset, + size, -EIO );return -EIO ; + }if (offset >0 )ifs_alloc (iter->inode, folio, iter->flags);
@@ -339,6 +350,10 @@ static void iomap_finish_folio_read(struct folio *folio, size_t off,spin_unlock_irqrestore (&ifs->state_lock, flags); } +if (error) +iomap_mapping_ioerror (folio->mapping,READ , +folio_pos (folio) + off, len, error); +if (finished)folio_end_read (folio, uptodate); }
@@ -558,11 +573,15 @@ static int iomap_read_folio_range(const struct iomap_iter *iter,const struct iomap *srcmap =iomap_iter_srcmap (iter);struct bio_vec bvec;struct bio bio; +int ret;bio_init (&bio, srcmap->bdev, &bvec,1 ,REQ_OP_READ ); bio.bi_iter.bi_sector =iomap_sector (srcmap, pos);bio_add_folio_nofail (&bio, folio, len,offset_in_folio (folio, pos)); -return submit_bio_wait (&bio); + ret =submit_bio_wait (&bio); +if (ret) +iomap_mapping_ioerror (folio->mapping,READ , pos, len, ret); +return ret; }#else static int iomap_read_folio_range(const struct iomap_iter *iter,
@@ -1674,6 +1693,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)u64 pos =folio_pos (folio);u64 end_pos = pos +folio_size (folio);u64 end_aligned =0 ; +loff_t orig_pos = pos;bool wb_pending = false;int error =0 ;u32 rlen;
@@ -1724,6 +1744,9 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)if (wb_pending) wpc->nr_folios++; +if (error && pos > orig_pos) +iomap_mapping_ioerror (inode->i_mapping,WRITE , orig_pos,0 , + error);/* * We can have dirty bits set past end of file in page_mkwrite path
fs/iomap/ioend.c +4 -0
@@ -55,6 +55,10 @@ static u32 iomap_finish_ioend_buffered(struct iomap_ioend *ioend)/* walk all folios in bio, ending page IO on them */ bio_for_each_folio_all (fi, bio) { +if (ioend->io_error) +iomap_mapping_ioerror (inode->i_mapping,WRITE , +folio_pos (fi.folio) + fi.offset, + fi.length, ioend->io_error);iomap_finish_folio_write (inode, fi.folio, fi.length); folio_count++; }
PATCH 8/19 iomap: report directio read and write errors to callers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add more hooks to report directio IO errors to the filesystem.
--- include/linux/iomap.h | 2 ++ fs/iomap/direct-io.c | 4 ++++ 2 files changed, 6 insertions(+)
include/linux/iomap.h +2 -0
@@ -486,6 +486,8 @@ struct iomap_dio_ops { unsigned flags);void (*submit_io)(const struct iomap_iter *iter,struct bio *bio,loff_t file_offset); +void (*ioerror)(struct inode *inode,int direction,loff_t pos, +u64 len,int error);/* * Filesystems wishing to attach private information to a direct io bio
fs/iomap/direct-io.c +4 -0
@@ -95,6 +95,10 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)if (dops && dops->end_io) ret = dops->end_io (iocb, dio->size, ret, dio->flags); +if (dio->error && dops && dops->ioerror) + dops->ioerror (file_inode (iocb->ki_filp), + (dio->flags &IOMAP_DIO_WRITE ) ?WRITE :READ , + offset, dio->size, dio->error);if (likely (!ret)) { ret = dio->size;
PATCH 9/19 xfs: create file io error hooks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create hooks within XFS to deliver IO errors to callers.
--- fs/xfs/xfs_file.h | 36 +++++++++++ fs/xfs/xfs_mount.h | 3 + fs/xfs/xfs_aops.c | 2 + fs/xfs/xfs_file.c | 167 ++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_super.c | 1 5 files changed, 208 insertions(+), 1 deletion(-)
fs/xfs/xfs_file.h +36 -0
@@ -12,4 +12,40 @@ extern const struct file_operations xfs_dir_file_operations;bool xfs_is_falloc_aligned (struct xfs_inode *ip,loff_t pos,long long int len); +enum xfs_file_ioerror_type { +XFS_FILE_IOERROR_BUFFERED_READ , +XFS_FILE_IOERROR_BUFFERED_WRITE , +XFS_FILE_IOERROR_DIRECT_READ , +XFS_FILE_IOERROR_DIRECT_WRITE , +}; + +struct xfs_file_ioerror_params { +xfs_ino_t ino; +loff_t pos; +u64 len; +u32 gen; +int error; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_file_ioerror_hook { +struct xfs_hook ioerror_hook; +}; + +void xfs_file_ioerror_hook_disable (void ); +void xfs_file_ioerror_hook_enable (void ); + +int xfs_file_ioerror_hook_add (struct xfs_mount *mp, +struct xfs_file_ioerror_hook *hook); +void xfs_file_ioerror_hook_del (struct xfs_mount *mp, +struct xfs_file_ioerror_hook *hook); +void xfs_file_ioerror_hook_setup (struct xfs_file_ioerror_hook *hook, +notifier_fn_t mod_fn); + +void xfs_vm_ioerror (struct address_space *mapping,int direction,loff_t pos, +u64 len,int error); +#else +# define xfs_vm_ioerror NULL +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_FILE_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -353,6 +353,9 @@ typedef struct xfs_mount {/* Hook to feed media error events to a daemon. */ struct xfs_hooks m_media_error_hooks; + +/* Hook to feed file io error events to a daemon. */ +struct xfs_hooks m_file_ioerror_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_aops.c +2 -0
@@ -22,6 +22,7 @@#include "xfs_icache.h" #include "xfs_zone_alloc.h" #include "xfs_rtgroup.h" +#include "xfs_file.h" struct xfs_writepage_ctx {struct iomap_writepage_ctx ctx;
@@ -810,6 +811,7 @@ const struct address_space_operations xfs_address_space_operations = { .is_partially_uptodate = iomap_is_partially_uptodate, .error_remove_folio = generic_error_remove_folio, .swap_activate = xfs_vm_swap_activate, + .ioerror = xfs_vm_ioerror, };const struct address_space_operations xfs_dax_aops = {
fs/xfs/xfs_file.c +166 -1
185 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2288,6 +2288,7 @@ xfs_init_fs_context(xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks);xfs_hooks_init (&mp->m_media_error_hooks); +xfs_hooks_init (&mp->m_file_ioerror_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 10/19 xfs: create a special file to pass filesystem health to userspace
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an ioctl that installs a file descriptor backed by an anon_inode file that will convey filesystem health events to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 8 ++ fs/xfs/xfs_healthmon.h | 16 +++++ fs/xfs/Kconfig | 8 ++ fs/xfs/Makefile | 1 fs/xfs/xfs_healthmon.c | 157 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 4 + 6 files changed, 194 insertions(+) create mode 100644 fs/xfs/xfs_healthmon.h create mode 100644 fs/xfs/xfs_healthmon.c
fs/xfs/libxfs/xfs_fs.h +8 -0
@@ -1003,6 +1003,13 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +struct xfs_health_monitor { +__u64 flags;/* flags */ +__u8 format;/* output format */ +__u8 pad1[7 ];/* zeroes */ +__u64 pad2[2 ];/* zeroes */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1042,6 +1049,7 @@ struct xfs_rtgroup_geometry {#define XFS_IOC_GETPARENTS_BY_HANDLE _IOWR('X', 63, struct xfs_getparents_by_handle)#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry) +#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (c) 2024-2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +#ifndef __XFS_HEALTHMON_H__ +#define __XFS_HEALTHMON_H__ + +#ifdef CONFIG_XFS_HEALTH_MONITOR +long xfs_ioc_health_monitor (struct xfs_mount *mp, +struct xfs_health_monitor __user *arg); +#else +# define xfs_ioc_health_monitor (mp, hmo) (-ENOTTY) +#endif /* CONFIG_XFS_HEALTH_MONITOR */ + +#endif /* __XFS_HEALTHMON_H__ */
fs/xfs/Kconfig +8 -0
@@ -121,6 +121,14 @@ config XFS_RT If unsure, say N. +config XFS_HEALTH_MONITOR + bool "Report filesystem health events to userspace" + depends on XFS_FS + select XFS_LIVE_HOOKS + default y + help + Report health events to userspace programs. + config XFS_DRAIN_INTENTS bool select JUMP_LABEL if HAVE_ARCH_JUMP_LABEL
fs/xfs/Makefile +1 -0
@@ -157,6 +157,7 @@ xfs-$(CONFIG_XFS_DRAIN_INTENTS) += xfs_drain.o xfs-$(CONFIG_XFS_LIVE_HOOKS) += xfs_hooks.o xfs-$(CONFIG_XFS_MEMORY_BUFS) += xfs_buf_mem.o xfs-$(CONFIG_XFS_BTREE_IN_MEM) += libxfs/xfs_btree_mem.o +xfs-$(CONFIG_XFS_HEALTH_MONITOR) += xfs_healthmon.o # online scrub/repair ifeq ($(CONFIG_XFS_ONLINE_SCRUB),y)
fs/xfs/xfs_healthmon.c new file +157 -0
157 lines not on this page · raw message
fs/xfs/xfs_ioctl.c +4 -0
@@ -41,6 +41,7 @@#include "xfs_exchrange.h" #include "xfs_handle.h" #include "xfs_rtgroup.h" +#include "xfs_healthmon.h" #include <linux/mount.h> #include <linux/fileattr.h>
@@ -1421,6 +1422,9 @@ xfs_file_ioctl(case XFS_IOC_COMMIT_RANGE :return xfs_ioc_commit_range (filp, arg); +case XFS_IOC_HEALTH_MONITOR : +return xfs_ioc_health_monitor (mp, arg); +default :return -ENOTTY ; }
PATCH 11/19 xfs: create event queuing, formatting, and discovery infrastructure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle.
Here, we've chosen json to export information to userspace. The structured key-value nature of json gives us enormous flexibility to modify the schema of what we'll send to userspace because we can add new keys at any time. Userspace can use whatever json parsers are available to consume the events and will not be confused by keys they don't recognize.
Note that we do NOT allow sending json back to the kernel, nor is there any intent to do that.
--- fs/xfs/libxfs/xfs_fs.h | 50 ++ fs/xfs/xfs_healthmon.h | 29 + fs/xfs/xfs_linux.h | 3 fs/xfs/xfs_trace.h | 171 +++++++ fs/xfs/libxfs/xfs_healthmon.schema.json | 129 +++++ fs/xfs/xfs_healthmon.c | 728 +++++++++++++++++++++++++++++++ fs/xfs/xfs_trace.c | 2 lib/seq_buf.c | 1 8 files changed, 1106 insertions(+), 7 deletions(-) create mode 100644 fs/xfs/libxfs/xfs_healthmon.schema.json
fs/xfs/libxfs/xfs_fs.h +50 -0
@@ -1003,6 +1003,45 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +/* Health monitor event domains */ + +/* affects the whole fs */ +#define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) + +/* Health monitor event types */ + +/* status of the monitor itself */ +#define XFS_HEALTH_MONITOR_TYPE_RUNNING (0) +#define XFS_HEALTH_MONITOR_TYPE_LOST (1) + +/* lost events */ +struct xfs_health_monitor_lost { +__u64 count; +}; + +struct xfs_health_monitor_event { +/* XFS_HEALTH_MONITOR_DOMAIN_* */ +__u32 domain; + +/* XFS_HEALTH_MONITOR_TYPE_* */ +__u32 type; + +/* Timestamp of the event, in nanoseconds since the Unix epoch */ +__u64 time_ns; + +/* +* Details of the event. The primary clients are written in python +* and rust, so break this up because bindgen hates anonymous structs +* and unions. +*/ +union { +struct xfs_health_monitor_lost lost; + } e; + +/* zeroes */ +__u64 pad[2 ]; +}; +struct xfs_health_monitor {__u64 flags;/* flags */ __u8 format;/* output format */
@@ -1010,6 +1049,17 @@ struct xfs_health_monitor {__u64 pad2[2 ];/* zeroes */ }; +/* Return all health status events, not just deltas */ +#define XFS_HEALTH_MONITOR_VERBOSE (1ULL << 0) + +#define XFS_HEALTH_MONITOR_ALL (XFS_HEALTH_MONITOR_VERBOSE) + +/* Return events in a C structure */ +#define XFS_HEALTH_MONITOR_FMT_CSTRUCT (0) + +/* Return events in JSON format */ +#define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* * ioctl commands that are used by Linux filesystems */
fs/xfs/xfs_healthmon.h +29 -0
@@ -6,6 +6,35 @@#ifndef __XFS_HEALTHMON_H__#define __XFS_HEALTHMON_H__ +enum xfs_healthmon_type { +XFS_HEALTHMON_RUNNING ,/* monitor running */ +XFS_HEALTHMON_LOST ,/* message lost */ +}; + +enum xfs_healthmon_domain { +XFS_HEALTHMON_MOUNT ,/* affects the whole fs */ +}; + +struct xfs_healthmon_event { +struct xfs_healthmon_event *next; + +enum xfs_healthmon_type type; +enum xfs_healthmon_domain domain; + +uint64_t time_ns; + +union { +/* lost events */ +struct { +uint64_t lostcount; + }; +/* mount */ +struct { +unsigned int flags; + }; + }; +}; +#ifdef CONFIG_XFS_HEALTH_MONITOR long xfs_ioc_health_monitor (struct xfs_mount *mp,struct xfs_health_monitor __user *arg);
fs/xfs/xfs_linux.h +3 -0
@@ -63,6 +63,9 @@ typedef __u32 xfs_nlink_t;#include <linux/xattr.h> #include <linux/mnt_idmapping.h> #include <linux/debugfs.h> +#ifdef CONFIG_XFS_HEALTH_MONITOR +# include <linux/seq_buf.h> +#endif #include <asm/page.h> #include <asm/div64.h>
fs/xfs/xfs_trace.h +171 -0
183 lines not on this page · raw message
fs/xfs/libxfs/xfs_healthmon.schema.json new file +129 -0
129 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +721 -7
799 lines not on this page · raw message
fs/xfs/xfs_trace.c +2 -0
@@ -51,6 +51,8 @@#include "xfs_rtgroup.h" #include "xfs_zone_alloc.h" #include "xfs_zone_priv.h" +#include "xfs_health.h" +#include "xfs_healthmon.h" /* * We include this last to have the helpers above available for the trace
lib/seq_buf.c +1 -0
@@ -245,6 +245,7 @@ int seq_buf_putmem(struct seq_buf *s, const void *mem, unsigned int len)seq_buf_set_overflow (s);return -1 ; } +EXPORT_SYMBOL_GPL (seq_buf_putmem);#define MAX_MEMHEX_BYTES 8U#define HEX_CHARS (MAX_MEMHEX_BYTES*2 + 1)
Re: [PATCH 11/19] xfs: create event queuing, formatting, and discovery infrastructure
Quoted 8 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle.
Quoted 6 lines
Here, we've chosen json to export information to userspace. The structured key-value nature of json gives us enormous flexibility to modify the schema of what we'll send to userspace because we can add new keys at any time. Userspace can use whatever json parsers are available to consume the events and will not be confused by keys they don't recognize.
Self-review: originally when I started designing this new subsystem, I wanted to explore data exchange formats that are more flexible and easier for humans to read than C structures. The thought being that when we want to rev (or worse, enlarge) the event format, it ought to be trivially easy to do that in a way that doesn't break old userspace.
I looked at formats such as protobufs and capnproto. These look really nice in that extending the wire format is fairly easy, you can give it a data schema and it generates the serialization code for you, handles endianness problems, etc. The huge downside is that neither support C all that well.
Too hard, and didn't want to port either of those huge sprawling libraries first to the kernel and then again to xfsprogs. Then I thought, how about JSON? Javascript objects are human readable, the kernel can emit json without much fuss (it's all just strings!) and there are plenty of interpreters for python/rust/c/etc.
There's a proposed schema format for json, which means that xfs can publish a description of the events that kernel will emit. Userspace consumers (e.g. xfsprogs/xfs_healer) can embed the same schema document and use it to validate the incoming events from the kernel, which means it can discard events that it doesn't understand, or garbage being emitted due to bugs.
However, json has a huge crutch -- javascript is well known for its vague definitions of what are numbers. This makes expressing a large number rather fraught, because the runtime is free to represent a number in nearly any way it wants. Stupider ones will truncate values to word size, others will roll out doubles for uint52_t (yes, fifty-two) with the resulting loss of precision. Not good when you're dealing with discrete units.
It just so happens that python's json library is smart enough to see a sequence of digits and put them in a u64 (at least on x86_64/aarch64) but an actual javascript interpreter (pasting into Firefox) isn't necessarily so clever.
It turns out that none of the proposed json schemas were ever ratified even in an open-consensus way, so json blobs are still just loosely structured blobs. The parsing in userspace was also noticeably slow and memory-consumptive.
As a result, I'm dropping all the json stuff from the codebase and leaving only the C structure event format. Since this is a mostly private interface, we can always rev the format in the traditional ways if we ever have to; there are 254 remaining unused format values.
I wanted to document the outcome of this experiment for posterity in a public place, and now I have done so.
--D
Quoted 1,303 lines
Note that we do NOT allow sending json back to the kernel, nor is there any intent to do that.
Signed-off-by "Darrick J. Wong" <[email protected]>--- fs/xfs/libxfs/xfs_fs.h | 50 ++ fs/xfs/xfs_healthmon.h | 29 + fs/xfs/xfs_linux.h | 3 fs/xfs/xfs_trace.h | 171 +++++++ fs/xfs/libxfs/xfs_healthmon.schema.json | 129 +++++ fs/xfs/xfs_healthmon.c | 728 +++++++++++++++++++++++++++++++ fs/xfs/xfs_trace.c | 2 lib/seq_buf.c | 1 8 files changed, 1106 insertions(+), 7 deletions(-) create mode 100644 fs/xfs/libxfs/xfs_healthmon.schema.jsonfs/xfs/libxfs/xfs_fs.h +50
-0@@ -1003,6 +1003,45 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +/* Health monitor event domains */ + +/* affects the whole fs */ +#define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) + +/* Health monitor event types */ + +/* status of the monitor itself */ +#define XFS_HEALTH_MONITOR_TYPE_RUNNING (0) +#define XFS_HEALTH_MONITOR_TYPE_LOST (1) + +/* lost events */ +struct xfs_health_monitor_lost { +__u64 count; +}; + +struct xfs_health_monitor_event { +/* XFS_HEALTH_MONITOR_DOMAIN_* */ +__u32 domain; + +/* XFS_HEALTH_MONITOR_TYPE_* */ +__u32 type; + +/* Timestamp of the event, in nanoseconds since the Unix epoch */ +__u64 time_ns; + +/* +* Details of the event. The primary clients are written in python +* and rust, so break this up because bindgen hates anonymous structs +* and unions. +*/ +union { +struct xfs_health_monitor_lost lost; + } e; + +/* zeroes */ +__u64 pad[2 ]; +}; +struct xfs_health_monitor {__u64 flags;/* flags */ __u8 format;/* output format */ @@ -1010,6 +1049,17 @@ struct xfs_health_monitor {__u64 pad2[2 ];/* zeroes */ }; +/* Return all health status events, not just deltas */ +#define XFS_HEALTH_MONITOR_VERBOSE (1ULL << 0) + +#define XFS_HEALTH_MONITOR_ALL (XFS_HEALTH_MONITOR_VERBOSE) + +/* Return events in a C structure */ +#define XFS_HEALTH_MONITOR_FMT_CSTRUCT (0) + +/* Return events in JSON format */ +#define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* * ioctl commands that are used by Linux filesystems */ fs/xfs/xfs_healthmon.h +29
-0@@ -6,6 +6,35 @@#ifndef __XFS_HEALTHMON_H__#define __XFS_HEALTHMON_H__ +enum xfs_healthmon_type { +XFS_HEALTHMON_RUNNING ,/* monitor running */ +XFS_HEALTHMON_LOST ,/* message lost */ +}; + +enum xfs_healthmon_domain { +XFS_HEALTHMON_MOUNT ,/* affects the whole fs */ +}; + +struct xfs_healthmon_event { +struct xfs_healthmon_event *next; + +enum xfs_healthmon_type type; +enum xfs_healthmon_domain domain; + +uint64_t time_ns; + +union { +/* lost events */ +struct { +uint64_t lostcount; + }; +/* mount */ +struct { +unsigned int flags; + }; + }; +}; +#ifdef CONFIG_XFS_HEALTH_MONITOR long xfs_ioc_health_monitor (struct xfs_mount *mp,struct xfs_health_monitor __user *arg);fs/xfs/xfs_linux.h +3
-0@@ -63,6 +63,9 @@ typedef __u32 xfs_nlink_t;#include <linux/xattr.h> #include <linux/mnt_idmapping.h> #include <linux/debugfs.h> +#ifdef CONFIG_XFS_HEALTH_MONITOR +# include <linux/seq_buf.h> +#endif #include <asm/page.h> #include <asm/div64.h> fs/xfs/xfs_trace.h +171
-0@@ -103,6 +103,8 @@ struct xfs_refcount_intent;struct xfs_metadir_update ;struct xfs_rtgroup ;struct xfs_open_zone ; +struct xfs_healthmon_event ; +struct xfs_health_update_params ;#define XFS_ATTR_FILTER_FLAGS \ { XFS_ATTR_ROOT, "ROOT" }, \@@ -5908,6 +5910,175 @@ DEFINE_EVENT(xfs_freeblocks_resv_class, name, \DEFINE_FREEBLOCKS_RESV_EVENT (xfs_freecounter_reserved);DEFINE_FREEBLOCKS_RESV_EVENT (xfs_freecounter_enospc); +#ifdef CONFIG_XFS_HEALTH_MONITOR +TRACE_EVENT (xfs_healthmon_lost_event, + TP_PROTO(const struct xfs_mount *mp, unsigned long long lost_prev), +TP_ARGS (mp, lost_prev), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(unsigned long long, lost_prev) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->lost_prev = lost_prev; + ), +TP_printk ("dev %d:%d lost_prev %llu" , +MAJOR (__entry->dev),MINOR (__entry->dev), + __entry->lost_prev) +); + +#define XFS_HEALTHMON_FLAGS_STRINGS \ + { XFS_HEALTH_MONITOR_VERBOSE, "verbose" } +#define XFS_HEALTHMON_FMT_STRINGS \ + { XFS_HEALTH_MONITOR_FMT_JSON, "json" }, \ + { XFS_HEALTH_MONITOR_FMT_CSTRUCT, "cstruct" } + +TRACE_EVENT (xfs_healthmon_create, + TP_PROTO(const struct xfs_mount *mp, u64 flags, u8 format), +TP_ARGS (mp, flags, format), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(u64, flags) +__field (u8, format) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->flags = flags; + __entry->format = format; + ), +TP_printk ("dev %d:%d flags %s format %s" , +MAJOR (__entry->dev),MINOR (__entry->dev), +__print_flags (__entry->flags,"|" ,XFS_HEALTHMON_FLAGS_STRINGS ), +__print_symbolic (__entry->format,XFS_HEALTHMON_FMT_STRINGS )) +); + +TRACE_EVENT (xfs_healthmon_copybuf, + TP_PROTO(const struct xfs_mount *mp, const struct iov_iter *iov, + const struct seq_buf *seqbuf, size_t outpos), +TP_ARGS (mp, iov, seqbuf, outpos), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(size_t, seqbuf_size) + __field(size_t, seqbuf_len) + __field(size_t, outpos) + __field(size_t, to_copy) +__field (size_t, iter_count) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->seqbuf_size = seqbuf->size; + __entry->seqbuf_len = seqbuf->len; + __entry->outpos = outpos; + __entry->to_copy = seqbuf->len - outpos; + __entry->iter_count = iov_iter_count(iov); + ), +TP_printk ("dev %d:%d seqsize %zu seqlen %zu out_pos %zu to_copy %zu iter_count %zu" , +MAJOR (__entry->dev),MINOR (__entry->dev), + __entry->seqbuf_size, + __entry->seqbuf_len, + __entry->outpos, + __entry->to_copy, + __entry->iter_count) +); + +DECLARE_EVENT_CLASS (xfs_healthmon_class, + TP_PROTO(const struct xfs_mount *mp, unsigned int events, + unsigned long long lost_prev), +TP_ARGS (mp, events, lost_prev), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(unsigned int, events) + __field(unsigned long long, lost_prev) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->events = events; + __entry->lost_prev = lost_prev; + ), +TP_printk ("dev %d:%d events %u lost_prev? %llu" , +MAJOR (__entry->dev),MINOR (__entry->dev), + __entry->events, + __entry->lost_prev) +); +#define DEFINE_HEALTHMON_EVENT (name) \ +DEFINE_EVENT(xfs_healthmon_class, name, \ + TP_PROTO(const struct xfs_mount *mp, unsigned int events, \ + unsigned long long lost_prev), \ + TP_ARGS(mp, events, lost_prev)) +DEFINE_HEALTHMON_EVENT (xfs_healthmon_read_start); +DEFINE_HEALTHMON_EVENT (xfs_healthmon_read_finish); +DEFINE_HEALTHMON_EVENT (xfs_healthmon_release); +DEFINE_HEALTHMON_EVENT (xfs_healthmon_unmount); + +#define XFS_HEALTHMON_TYPE_STRINGS \ + { XFS_HEALTHMON_LOST, "lost" } + +#define XFS_HEALTHMON_DOMAIN_STRINGS \ + { XFS_HEALTHMON_MOUNT, "mount" } + +TRACE_DEFINE_ENUM (XFS_HEALTHMON_LOST ); + +TRACE_DEFINE_ENUM (XFS_HEALTHMON_MOUNT ); + +DECLARE_EVENT_CLASS (xfs_healthmon_event_class, + TP_PROTO(const struct xfs_mount *mp, const struct xfs_healthmon_event *event), +TP_ARGS (mp, event), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(unsigned int, type) + __field(unsigned int, domain) + __field(unsigned int, mask) + __field(unsigned long long, ino) + __field(unsigned int, gen) + __field(unsigned int, group) + __field(unsigned long long, offset) + __field(unsigned long long, length) + __field(unsigned long long, lostcount) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->type = event->type; + __entry->domain = event->domain; + __entry->mask = 0; + __entry->group = 0; + __entry->ino = 0; + __entry->gen = 0; + __entry->offset = 0; + __entry->length = 0; + __entry->lostcount = 0; + switch (__entry->domain) { +case XFS_HEALTHMON_MOUNT : +switch (__entry->type) { +case XFS_HEALTHMON_LOST : + __entry->lostcount = event->lostcount; +break ; + } +break ; + } + ), +TP_printk ("dev %d:%d type %s domain %s mask 0x%x ino 0x%llx gen 0x%x offset 0x%llx len 0x%llx group 0x%x lost %llu" , +MAJOR (__entry->dev),MINOR (__entry->dev), +__print_symbolic (__entry->type,XFS_HEALTHMON_TYPE_STRINGS ), +__print_symbolic (__entry->domain,XFS_HEALTHMON_DOMAIN_STRINGS ), + __entry->mask, + __entry->ino, + __entry->gen, + __entry->offset, + __entry->length, + __entry->group, + __entry->lostcount) +); +#define DEFINE_HEALTHMONEVENT_EVENT (name) \ +DEFINE_EVENT(xfs_healthmon_event_class, name, \ + TP_PROTO(const struct xfs_mount *mp, const struct xfs_healthmon_event *event), \ + TP_ARGS(mp, event)) +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_push); +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_pop); +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_format); +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_format_overflow); +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_drop); +#endif /* CONFIG_XFS_HEALTH_MONITOR */ +#endif /* _TRACE_XFS_H */ #undef TRACE_INCLUDE_PATHfs/xfs/libxfs/xfs_healthmon.schema.json new file +129
-0@@ -0,0 +1,129 @@ +{ + "$comment": [ + "SPDX-License-Identifier: GPL-2.0-or-later", + "Copyright (c) 2024-2025 Oracle. All Rights Reserved.", + "Author: Darrick J. Wong <[email protected]>", + "", + "This schema file describes the format of the json objects", + "readable from the fd returned by the XFS_IOC_HEALTHMON", + "ioctl." + ], + + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/fs/xfs/libxfs/xfs_healthmon.schema.json", + + "title": "XFS Health Monitoring Events", + + "$comment": "Events must be one of the following types:", + "oneOf": [ + { + "$ref": "#/$events/running" + }, + { + "$ref": "#/$events/unmount" + }, + { + "$ref": "#/$events/lost" + } + ], + + "$comment": "Simple data types are defined here.", + "$defs": { + "time_ns": { + "title": "Time of Event", + "description": "Timestamp of the event, in nanoseconds since the Unix epoch.", + "type": "integer" + }, + "count": { + "title": "Count of events", + "description": "Number of events.", + "type": "integer", + "minimum": 1 + } + }, + + "$comment": "Event types are defined here.", + "$events": { + "running": { + "title": "Health Monitoring Running", + "$comment": [ + "The health monitor is actually running." + ], + "type": "object", + + "properties": { + "type": { + "const": "running" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "mount" + } + }, + + "required": [ + "type", + "time_ns", + "domain" + ] + }, + "unmount": { + "title": "Filesystem Unmounted", + "$comment": [ + "The filesystem was unmounted." + ], + "type": "object", + + "properties": { + "type": { + "const": "unmount" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "mount" + } + }, + + "required": [ + "type", + "time_ns", + "domain" + ] + }, + "lost": { + "title": "Health Monitoring Events Lost", + "$comment": [ + "Previous health monitoring events were", + "dropped due to memory allocation failures", + "or queue limits." + ], + "type": "object", + + "properties": { + "type": { + "const": "lost" + }, + "count": { + "$ref": "#/$defs/count" + }, + "time_ns": { + "$ref": "#/$defs/time_ns" + }, + "domain": { + "const": "mount" + } + }, + + "required": [ + "type", + "count", + "time_ns", + "domain" + ] + } + } +}fs/xfs/xfs_healthmon.c +721
-7@@ -40,12 +40,558 @@* so that the queueing and processing of the events do not pin the mount and * cannot slow down the main filesystem. The healthmon object can exist past * the end of the filesystem mount. +* +* Please see the xfs_healthmon.schema.json file for a description of the +* format of the json events that are conveyed to userspace. */ +/* Allow this many events to build up in memory per healthmon fd. */ +#define XFS_HEALTHMON_MAX_EVENTS \ + (32768 / sizeof(struct xfs_healthmon_event)) + +struct flag_string { +unsigned int mask; +const char *str; +}; +struct xfs_healthmon { +/* lock for mp and eventlist */ +struct mutex lock; + +/* waiter for signalling the arrival of events */ +struct wait_queue_head wait; + +/* list of event objects */ +struct xfs_healthmon_event *first_event; +struct xfs_healthmon_event *last_event; +struct xfs_mount *mp; + +/* number of events */ +unsigned int events; + +/* +* Buffer for formatting events. New buffer data are appended to the +* end of the seqbuf, and outpos is used to determine where to start +* a copy_iter. Both are protected by inode_lock. +*/ +struct seq_buf outbuf; +size_t outpos; + +/* XFS_HEALTH_MONITOR_FMT_* */ +uint8_t format; + +/* do we want all events? */ +bool verbose; + +/* did we lose previous events? */ +unsigned long long lost_prev_event; + +/* total counts of events observed and lost events */ +unsigned long long total_events; +unsigned long long total_lost; }; +static inline void xfs_healthmon_bump_events (struct xfs_healthmon *hm) +{ + hm->events++; + hm->total_events++; +} + +static inline void xfs_healthmon_bump_lost (struct xfs_healthmon *hm) +{ + hm->lost_prev_event++; + hm->total_lost++; +} + +/* Remove an event from the head of the list. */ +static inline int +xfs_healthmon_free_head ( +struct xfs_healthmon *hm, +struct xfs_healthmon_event *event) +{ +struct xfs_healthmon_event *head; + +mutex_lock (&hm->lock); + head = hm->first_event; +if (head != event) { +ASSERT (hm->first_event == event); +mutex_unlock (&hm->lock); +return -EFSCORRUPTED ; + } + +if (hm->last_event == head) + hm->last_event =NULL ; + hm->first_event = head->next; + hm->events--; +mutex_unlock (&hm->lock); + +trace_xfs_healthmon_pop (hm->mp, head); +kfree (event); +return 0 ; +} + +/* Push an event onto the end of the list. */ +static inline void +__xfs_healthmon_push ( +struct xfs_healthmon *hm, +struct xfs_healthmon_event *event) +{ +if (!hm->first_event) + hm->first_event = event; +if (hm->last_event) + hm->last_event->next = event; + hm->last_event = event; + event->next =NULL ; +xfs_healthmon_bump_events (hm); +wake_up (&hm->wait); + +trace_xfs_healthmon_push (hm->mp, event); +} + +/* Push an event onto the end of the list if we're not full. */ +static inline int +xfs_healthmon_push ( +struct xfs_healthmon *hm, +struct xfs_healthmon_event *event) +{ +if (hm->events >=XFS_HEALTHMON_MAX_EVENTS ) { +trace_xfs_healthmon_lost_event (hm->mp, hm->lost_prev_event); + +xfs_healthmon_bump_lost (hm); +return -ENOMEM ; + } + +__xfs_healthmon_push (hm, event); +return 0 ; +} + +/* Create a new event or record that we failed. */ +static struct xfs_healthmon_event * +xfs_healthmon_alloc ( +struct xfs_healthmon *hm, +enum xfs_healthmon_type type, +enum xfs_healthmon_domain domain) +{ +struct timespec64 now; +struct xfs_healthmon_event *event; + + event =kzalloc (sizeof (*event),GFP_NOFS ); +if (!event) { +trace_xfs_healthmon_lost_event (hm->mp, hm->lost_prev_event); + +xfs_healthmon_bump_lost (hm); +return NULL ; + } + + event->type = type; + event->domain = domain; +ktime_get_coarse_real_ts64 (&now); + event->time_ns = (now.tv_sec *NSEC_PER_SEC ) + now.tv_nsec; + +return event; +} + +/* +* Before we accept an event notification from a live update hook, we need to +* clear out any previously lost events. +*/ +static inline int +xfs_healthmon_start_live_update ( +struct xfs_healthmon *hm) +{ +struct xfs_healthmon_event *event; + +/* If the queue is already full.... */ +if (hm->events >=XFS_HEALTHMON_MAX_EVENTS ) { +trace_xfs_healthmon_lost_event (hm->mp, hm->lost_prev_event); + +if (hm->last_event && + hm->last_event->type ==XFS_HEALTHMON_LOST ) { +/* +* ...and the last event notes lost events, then add +* the number of events we already lost, plus one for +* this event that we're about to lose. +*/ + hm->last_event->lostcount += hm->lost_prev_event +1 ; + hm->lost_prev_event =0 ; + }else { +/* +* ...try to create a new lost event. Add the number +* of events we previously lost, plus one for this +* event. +*/ + event =xfs_healthmon_alloc (hm,XFS_HEALTHMON_LOST , +XFS_HEALTHMON_MOUNT ); +if (!event) { +xfs_healthmon_bump_lost (hm); +return -ENOMEM ; + } + event->lostcount = hm->lost_prev_event +1 ; + hm->lost_prev_event =0 ; + +__xfs_healthmon_push (hm, event); + } + +return -ENOSPC ; + } + +/* If we lost an event in the past, but the queue isn't yet full... */ +if (hm->lost_prev_event) { +/* +* ...try to create a new lost event. Add the number of events +* we previously lost, plus one for this event. +*/ + event =xfs_healthmon_alloc (hm,XFS_HEALTHMON_LOST , +XFS_HEALTHMON_MOUNT ); +if (!event) { +xfs_healthmon_bump_lost (hm); +return -ENOMEM ; + } + event->lostcount = hm->lost_prev_event; + hm->lost_prev_event =0 ; + +/* +* If adding this lost event pushes us over the limit, we're +* going to lose the current event. Note that in the lost +* event count too. +*/ +if (hm->events ==XFS_HEALTHMON_MAX_EVENTS -1 ) + event->lostcount++; + +__xfs_healthmon_push (hm, event); +if (hm->events >=XFS_HEALTHMON_MAX_EVENTS ) { +trace_xfs_healthmon_lost_event (hm->mp, + hm->lost_prev_event); +return -ENOSPC ; + } + } + +/* +* The queue is not full and it is not currently the case that events +* were lost. +*/ +return 0 ; +} + +/* Render the health update type as a string. */ +STATIC const char * +xfs_healthmon_typestring ( +const struct xfs_healthmon_event *event) +{ +static const char *type_strings[] = { + [XFS_HEALTHMON_RUNNING ] ="running" , + [XFS_HEALTHMON_LOST ] ="lost" , + }; + +if (event->type >=ARRAY_SIZE (type_strings)) +return "?" ; + +return type_strings[event->type]; +} + +/* Render the health domain as a string. */ +STATIC const char * +xfs_healthmon_domstring ( +const struct xfs_healthmon_event *event) +{ +static const char *dom_strings[] = { + [XFS_HEALTHMON_MOUNT ] ="mount" , + }; + +if (event->domain >=ARRAY_SIZE (dom_strings)) +return "?" ; + +return dom_strings[event->domain]; +} + +/* Convert a flags bitmap into a jsonable string. */ +static inline int +xfs_healthmon_format_flags ( +struct seq_buf *outbuf, +const struct flag_string *strings, +size_t nr_strings, +unsigned int flags) +{ +const struct flag_string *p; +ssize_t ret; +unsigned int i; +bool first = true; + +for (i =0 , p = strings; i < nr_strings; i++, p++) { +if (!(p->mask & flags)) +continue ; + + ret =seq_buf_printf (outbuf,"%s\"%s\"" , + first ?"" :", " , p->str); +if (ret <0 ) +return ret; + + first = false; + flags &= ~p->mask; + } + +for (i =0 ; flags !=0 && i <sizeof (flags) *NBBY ; i++) { +if (!(flags & (1U << i))) +continue ; + +/* json doesn't support hexadecimal notation */ + ret =seq_buf_printf (outbuf,"%s%u" , + first ?"" :", " , (1U << i)); +if (ret <0 ) +return ret; + + first = false; + } + +return 0 ; +} + +/* Convert the event mask into a jsonable string. */ +static inline int +__xfs_healthmon_format_mask ( +struct seq_buf *outbuf, +const char *descr, +const struct flag_string *strings, +size_t nr_strings, +unsigned int mask) +{ +ssize_t ret; + + ret =seq_buf_printf (outbuf," \"%s\": [" , descr); +if (ret <0 ) +return ret; + + ret =xfs_healthmon_format_flags (outbuf, strings, nr_strings, mask); +if (ret <0 ) +return ret; + +return seq_buf_printf (outbuf,"],\n" ); +} + +#define xfs_healthmon_format_mask (o, d, s, m) \ + __xfs_healthmon_format_mask((o), (d), (s), ARRAY_SIZE(s), (m)) + +static inline void +xfs_healthmon_reset_outbuf ( +struct xfs_healthmon *hm) +{ + hm->outpos =0 ; +seq_buf_clear (&hm->outbuf); +} + +/* Render lost event mask as a string set */ +static int +xfs_healthmon_format_lost ( +struct seq_buf *outbuf, +const struct xfs_healthmon_event *event) +{ +return seq_buf_printf (outbuf," \"count\": %llu,\n" , + event->lostcount); +} + +/* +* Format an event into json. Returns 0 if we formatted the event. If +* formatting the event overflows the buffer, returns -1 with the seqbuf len +* unchanged. +*/ +STATIC int +xfs_healthmon_format_json ( +struct xfs_healthmon *hm, +const struct xfs_healthmon_event *event) +{ +struct seq_buf *outbuf = &hm->outbuf; +size_t old_seqlen = outbuf->len; +int ret; + +trace_xfs_healthmon_format (hm->mp, event); + + ret =seq_buf_printf (outbuf,"{\n" ); +if (ret <0 ) + goto overrun; + + ret =seq_buf_printf (outbuf," \"domain\": \"%s\",\n" , +xfs_healthmon_domstring (event)); +if (ret <0 ) + goto overrun; + + ret =seq_buf_printf (outbuf," \"type\": \"%s\",\n" , +xfs_healthmon_typestring (event)); +if (ret <0 ) + goto overrun; + +switch (event->domain) { +case XFS_HEALTHMON_MOUNT : +switch (event->type) { +case XFS_HEALTHMON_RUNNING : +/* nothing to format */ +break ; +case XFS_HEALTHMON_LOST : + ret =xfs_healthmon_format_lost (outbuf, event); +break ; +default : +break ; + } +break ; + } +if (ret <0 ) + goto overrun; + +/* The last element in the json must not have a trailing comma. */ + ret =seq_buf_printf (outbuf," \"time_ns\": %llu\n" , + event->time_ns); +if (ret <0 ) + goto overrun; + + ret =seq_buf_printf (outbuf,"}\n" ); +if (ret <0 ) + goto overrun; + +ASSERT (!seq_buf_has_overflowed (outbuf)); +return 0 ; +overrun: +/* +* We overflowed the buffer and could not format the event. Reset the +* seqbuf and tell the caller not to delete the event. +*/ +trace_xfs_healthmon_format_overflow (hm->mp, event); + outbuf->len = old_seqlen; +return -1 ; +} + +static const unsigned int domain_map[] = { + [XFS_HEALTHMON_MOUNT ] =XFS_HEALTH_MONITOR_DOMAIN_MOUNT , +}; + +static const unsigned int type_map[] = { + [XFS_HEALTHMON_RUNNING ] =XFS_HEALTH_MONITOR_TYPE_RUNNING , + [XFS_HEALTHMON_LOST ] =XFS_HEALTH_MONITOR_TYPE_LOST , +}; + +/* Render event as a C structure */ +STATIC int +xfs_healthmon_format_cstruct ( +struct xfs_healthmon *hm, +const struct xfs_healthmon_event *event) +{ +struct xfs_health_monitor_event hme = { + .time_ns = event->time_ns, + }; +struct seq_buf *outbuf = &hm->outbuf; +size_t old_seqlen = outbuf->len; +int ret; + +trace_xfs_healthmon_format (hm->mp, event); + +if (event->domain <0 || event->domain >=ARRAY_SIZE (domain_map) || + event->type <0 || event->type >=ARRAY_SIZE (type_map)) +return -EFSCORRUPTED ; + + hme.domain = domain_map[event->domain]; + hme.type = type_map[event->type]; + +/* fill in the event-specific details */ +switch (event->domain) { +case XFS_HEALTHMON_MOUNT : +switch (event->type) { +case XFS_HEALTHMON_LOST : + hme.e.lost.count = event->lostcount; +break ; +default : +break ; + } +break ; +default : +break ; + } + + ret =seq_buf_putmem (outbuf, &hme,sizeof (hme)); +if (ret <0 ) { +/* +* We overflowed the buffer and could not format the event. +* Reset the seqbuf and tell the caller not to delete the +* event. +*/ +trace_xfs_healthmon_format_overflow (hm->mp, event); + outbuf->len = old_seqlen; +return -1 ; + } + +ASSERT (!seq_buf_has_overflowed (outbuf)); +return 0 ; +} + +/* How many bytes are waiting in the outbuf to be copied? */ +static inline size_t +xfs_healthmon_outbuf_bytes ( +struct xfs_healthmon *hm) +{ +unsigned int used =seq_buf_used (&hm->outbuf); + +if (used > hm->outpos) +return used - hm->outpos; +return 0 ; +} + +/* +* Do we have something for userspace to do? This can mean unmount events, +* events pending in the queue, or pending bytes in the outbuf. +*/ +static inline bool +xfs_healthmon_has_eventdata ( +struct xfs_healthmon *hm) +{ +return hm->events >0 ||xfs_healthmon_outbuf_bytes (hm) >0 ; +} + +/* Try to copy the rest of the outbuf to the iov iter. */ +STATIC ssize_t +xfs_healthmon_copybuf ( +struct xfs_healthmon *hm, +struct iov_iter *to) +{ +size_t to_copy; +size_t w =0 ; + +trace_xfs_healthmon_copybuf (hm->mp, to, &hm->outbuf, hm->outpos); + + to_copy =xfs_healthmon_outbuf_bytes (hm); +if (to_copy) { + w =copy_to_iter (hm->outbuf.buffer + hm->outpos, to_copy, to); +if (!w) +return -EFAULT ; + + hm->outpos += w; + } + +/* +* Nothing left to copy? Reset the seqbuf pointers and outbuf to the +* start since there's no live data in the buffer. +*/ +if (xfs_healthmon_outbuf_bytes (hm) ==0 ) +xfs_healthmon_reset_outbuf (hm); +return w; +} + +/* +* See if there's an event waiting for us. If the fs is no longer mounted, +* don't bother sending any more events. +*/ +static inline struct xfs_healthmon_event * +xfs_healthmon_peek ( +struct xfs_healthmon *hm) +{ +struct xfs_healthmon_event *event; + +mutex_lock (&hm->lock); +if (hm->mp) + event = hm->first_event; +else + event =NULL ; +mutex_unlock (&hm->lock); +return event; +} +/* * Convey queued event data to userspace. First copy any remaining bytes in * the outbuf, then format the oldest event into the outbuf and copy that too. @@ -55,7 +601,125 @@ xfs_healthmon_read_iter(struct kiocb *iocb, struct iov_iter *to) { - return -EIO; + struct file *file = iocb->ki_filp; +struct inode *inode =file_inode (file); +struct xfs_healthmon *hm = file->private_data; +struct xfs_healthmon_event *event; +size_t copied =0 ; +ssize_t ret =0 ; + +/* Wait for data to become available */ +if (!(file->f_flags &O_NONBLOCK )) { + ret =wait_event_interruptible (hm->wait, +xfs_healthmon_has_eventdata (hm)); +if (ret) +return ret; + }else if (!xfs_healthmon_has_eventdata (hm)) { +return -EAGAIN ; + } + +/* Allocate formatting buffer up to 64k if necessary */ +if (hm->outbuf.size ==0 ) { +void *outbuf; +size_t bufsize =min (65536 ,max (PAGE_SIZE , +iov_iter_count (to))); + + outbuf =kzalloc (bufsize,GFP_KERNEL ); +if (!outbuf) { + bufsize =PAGE_SIZE ; + outbuf =kzalloc (bufsize,GFP_KERNEL ); +if (!outbuf) +return -ENOMEM ; + } + +inode_lock (inode); +if (hm->outbuf.size ==0 ) { +seq_buf_init (&hm->outbuf, outbuf, bufsize); + hm->outpos =0 ; + }else { +kfree (outbuf); + } + }else { +inode_lock (inode); + } + +trace_xfs_healthmon_read_start (hm->mp, hm->events, hm->lost_prev_event); + +/* +* If there's anything left in the seqbuf, copy that before formatting +* more events. +*/ + ret =xfs_healthmon_copybuf (hm, to); +if (ret <0 ) + goto out_unlock; + copied += ret; + +while (iov_iter_count (to) >0 ) { +/* Format the next events into the outbuf until it's full. */ +while ((event =xfs_healthmon_peek (hm)) !=NULL ) { +switch (hm->format) { +case XFS_HEALTH_MONITOR_FMT_JSON : + ret =xfs_healthmon_format_json (hm, event); +break ; +case XFS_HEALTH_MONITOR_FMT_CSTRUCT : + ret =xfs_healthmon_format_cstruct (hm, event); +break ; +default : + ret = -EINVAL ; + goto out_unlock; + } +if (ret <0 ) +break ; + ret =xfs_healthmon_free_head (hm, event); +if (ret) + goto out_unlock; + } + +/* Copy it to userspace */ + ret =xfs_healthmon_copybuf (hm, to); +if (ret <=0 ) +break ; + + copied += ret; + } + +out_unlock: +trace_xfs_healthmon_read_finish (hm->mp, hm->events, hm->lost_prev_event); +inode_unlock (inode); +return copied ?: ret; +} + +/* Poll for available events. */ +STATIC __poll_t +xfs_healthmon_poll ( +struct file *file, +struct poll_table_struct *wait) +{ +struct xfs_healthmon *hm = file->private_data; +__poll_t mask =0 ; + +poll_wait (file, &hm->wait, wait); + +if (xfs_healthmon_has_eventdata (hm)) + mask |=EPOLLIN ; +return mask; +} + +/* Free all events */ +STATIC void +xfs_healthmon_free_events ( +struct xfs_healthmon *hm) +{ +struct xfs_healthmon_event *event, *next; + + event = hm->first_event; +while (event !=NULL ) { +trace_xfs_healthmon_drop (hm->mp, event); + next = event->next; +kfree (event); + event = next; + } + hm->first_event = hm->last_event =NULL ; }/* Free the health monitoring information. */ @@ -66,6 +730,14 @@ xfs_healthmon_release( {struct xfs_healthmon *hm = file->private_data; +trace_xfs_healthmon_release (hm->mp, hm->events, hm->lost_prev_event); + +wake_up_all (&hm->wait); + +mutex_destroy (&hm->lock); +xfs_healthmon_free_events (hm); +if (hm->outbuf.size) +kfree (hm->outbuf.buffer);kfree (hm);return 0 ;@@ -76,9 +748,10 @@ static inline boolxfs_healthmon_validate ( const struct xfs_health_monitor *hmo) { -if (hmo->flags) +if (hmo->flags & ~XFS_HEALTH_MONITOR_ALL )return false; -if (hmo->format) +if (hmo->format !=XFS_HEALTH_MONITOR_FMT_JSON && + hmo->format !=XFS_HEALTH_MONITOR_FMT_CSTRUCT )return false;if (memchr_inv (&hmo->pad1,0 ,sizeof (hmo->pad1)))return false;@@ -89,6 +762,19 @@ xfs_healthmon_validate(/* Emit some data about the health monitoring fd. */ #ifdef CONFIG_PROC_FS +static const char * +xfs_healthmon_format_string (const struct xfs_healthmon *hm) +{ +switch (hm->format) { +case XFS_HEALTH_MONITOR_FMT_JSON : +return "json" ; +case XFS_HEALTH_MONITOR_FMT_CSTRUCT : +return "blob" ; + } + +return "" ; +} + static void xfs_healthmon_show_fdinfo( struct seq_file *m,@@ -96,8 +782,13 @@ xfs_healthmon_show_fdinfo( {struct xfs_healthmon *hm = file->private_data; -seq_printf (m,"state:\talive\ndev:\t%s\n" , - hm->mp->m_super->s_id); +mutex_lock (&hm->lock); +seq_printf (m,"state:\talive\ndev:\t%s\nformat:\t%s\nevents:\t%llu\nlost:\t%llu\n" , + hm->mp->m_super->s_id, +xfs_healthmon_format_string (hm), + hm->total_events, + hm->total_lost); +mutex_unlock (&hm->lock); }#endif @@ -107,6 +798,7 @@ static const struct file_operations xfs_healthmon_fops = { .show_fdinfo = xfs_healthmon_show_fdinfo, #endif .read_iter = xfs_healthmon_read_iter, + .poll = xfs_healthmon_poll, .release = xfs_healthmon_release, };@@ -121,6 +813,7 @@ xfs_ioc_health_monitor( {struct xfs_health_monitor hmo;struct xfs_healthmon *hm; +struct xfs_healthmon_event *event;int fd;int ret;@@ -137,6 +830,23 @@ xfs_ioc_health_monitor(if (!hm)return -ENOMEM ; hm->mp = mp; + hm->format = hmo.format; + +seq_buf_init (&hm->outbuf,NULL ,0 ); +mutex_init (&hm->lock); +init_waitqueue_head (&hm->wait); + +if (hmo.flags &XFS_HEALTH_MONITOR_VERBOSE ) + hm->verbose = true; + +/* Queue up the first event that lets the client know we're running. */ + event =xfs_healthmon_alloc (hm,XFS_HEALTHMON_RUNNING , +XFS_HEALTHMON_MOUNT ); +if (!event) { + ret = -ENOMEM ; + goto out_mutex; + } +__xfs_healthmon_push (hm, event);/* * Create the anonymous file. If it succeeds, the file owns hm and @@ -146,12 +856,16 @@ xfs_ioc_health_monitor( O_CLOEXEC | O_RDONLY);if (fd <0 ) { ret = fd; - goto out_hm; + goto out_mutex; } +trace_xfs_healthmon_create (mp, hmo.flags, hmo.format); +return fd; -out_hm: +out_mutex: +mutex_destroy (&hm->lock); +xfs_healthmon_free_events (hm);kfree (hm);return ret; }fs/xfs/xfs_trace.c +2
-0@@ -51,6 +51,8 @@#include "xfs_rtgroup.h" #include "xfs_zone_alloc.h" #include "xfs_zone_priv.h" +#include "xfs_health.h" +#include "xfs_healthmon.h" /* * We include this last to have the helpers above available for the trace lib/seq_buf.c +1
-0@@ -245,6 +245,7 @@ int seq_buf_putmem(struct seq_buf *s, const void *mem, unsigned int len)seq_buf_set_overflow (s);return -1 ; } +EXPORT_SYMBOL_GPL (seq_buf_putmem);#define MAX_MEMHEX_BYTES 8U#define HEX_CHARS (MAX_MEMHEX_BYTES*2 + 1)
PATCH 12/19 xfs: report metadata health events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a metadata health event hook so that we can send events to userspace as we collect information. The unmount hook severs the weak reference between the health monitor and the filesystem it's monitoring; when this happens, we stop reporting events because there's no longer any point.
--- fs/xfs/libxfs/xfs_fs.h | 38 +++ fs/xfs/libxfs/xfs_health.h | 5 fs/xfs/xfs_healthmon.h | 31 ++ fs/xfs/xfs_trace.h | 98 ++++++- fs/xfs/libxfs/xfs_healthmon.schema.json | 315 +++++++++++++++++++++ fs/xfs/xfs_health.c | 67 ++++ fs/xfs/xfs_healthmon.c | 465 +++++++++++++++++++++++++++++++ 7 files changed, 1010 insertions(+), 9 deletions(-)
fs/xfs/libxfs/xfs_fs.h +38 -0
@@ -1008,17 +1008,52 @@ struct xfs_rtgroup_geometry {/* affects the whole fs */ #define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FS (1) +#define XFS_HEALTH_MONITOR_DOMAIN_AG (2) +#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3) +#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* Health monitor event types */ /* status of the monitor itself */ #define XFS_HEALTH_MONITOR_TYPE_RUNNING (0)#define XFS_HEALTH_MONITOR_TYPE_LOST (1) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_TYPE_SICK (2) +#define XFS_HEALTH_MONITOR_TYPE_CORRUPT (3) +#define XFS_HEALTH_MONITOR_TYPE_HEALTHY (4) + +/* filesystem was unmounted */ +#define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* lost events */ struct xfs_health_monitor_lost {__u64 count; }; +/* fs/rt metadata */ +struct xfs_health_monitor_fs { +/* XFS_FSOP_GEOM_SICK_* flags */ +__u32 mask; +}; + +/* ag/rtgroup metadata */ +struct xfs_health_monitor_group { +/* XFS_{AG,RTGROUP}_SICK_* flags */ +__u32 mask; +__u32 gno; +}; + +/* inode metadata */ +struct xfs_health_monitor_inode { +/* XFS_BS_SICK_* flags */ +__u32 mask; +__u32 gen; +__u64 ino; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1036,6 +1071,9 @@ struct xfs_health_monitor_event {*/ union {struct xfs_health_monitor_lost lost; +struct xfs_health_monitor_fs fs; +struct xfs_health_monitor_group group; +struct xfs_health_monitor_inode inode; } e;/* zeroes */
fs/xfs/libxfs/xfs_health.h +5 -0
@@ -336,4 +336,9 @@ void xfs_health_hook_del(struct xfs_mount *mp, struct xfs_health_hook *hook);void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn);#endif /* CONFIG_XFS_LIVE_HOOKS */ +unsigned int xfs_healthmon_inode_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_rtgroup_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_perag_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_fs_mask (unsigned int sick_mask); +#endif /* __XFS_HEALTH_H__ */
fs/xfs/xfs_healthmon.h +31 -0
@@ -9,10 +9,23 @@enum xfs_healthmon_type {XFS_HEALTHMON_RUNNING ,/* monitor running */ XFS_HEALTHMON_LOST ,/* message lost */ +XFS_HEALTHMON_UNMOUNT ,/* filesystem is unmounting */ + +/* metadata health events */ +XFS_HEALTHMON_SICK ,/* runtime corruption observed */ +XFS_HEALTHMON_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHMON_HEALTHY ,/* fsck reported healthy structure */ + };enum xfs_healthmon_domain {XFS_HEALTHMON_MOUNT ,/* affects the whole fs */ + +/* metadata health events */ +XFS_HEALTHMON_FS ,/* main filesystem metadata */ +XFS_HEALTHMON_AG ,/* allocation group metadata */ +XFS_HEALTHMON_INODE ,/* inode metadata */ +XFS_HEALTHMON_RTGROUP ,/* realtime group metadata */ };struct xfs_healthmon_event {
@@ -32,6 +45,24 @@ struct xfs_healthmon_event {struct {unsigned int flags; }; +/* fs/rt metadata */ +struct { +/* XFS_SICK_* flags */ +unsigned int fsmask; + }; +/* ag/rtgroup metadata */ +struct { +/* XFS_SICK_* flags */ +unsigned int grpmask; +unsigned int group; + }; +/* inode metadata */ +struct { +/* XFS_SICK_INO_* flags */ +unsigned int imask; +uint32_t gen; +xfs_ino_t ino; + }; }; };
fs/xfs/xfs_trace.h +96 -2
127 lines not on this page · raw message
fs/xfs/libxfs/xfs_healthmon.schema.json +315 -0
333 lines not on this page · raw message
fs/xfs/xfs_health.c +67 -0
91 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +458 -7
609 lines not on this page · raw message
PATCH 13/19 xfs: report shutdown events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a shutdown hook so that we can send notifications to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 18 +++++ fs/xfs/xfs_healthmon.h | 5 + fs/xfs/xfs_trace.h | 28 +++++++ fs/xfs/libxfs/xfs_healthmon.schema.json | 62 ++++++++++++++++ fs/xfs/xfs_healthmon.c | 119 ++++++++++++++++++++++++++++++- 5 files changed, 229 insertions(+), 3 deletions(-)
fs/xfs/libxfs/xfs_fs.h +18 -0
@@ -1028,6 +1028,9 @@ struct xfs_rtgroup_geometry {/* filesystem was unmounted */ #define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* filesystem shutdown */ +#define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1054,6 +1057,20 @@ struct xfs_health_monitor_inode {__u64 ino; }; +/* shutdown reasons */ +#define XFS_HEALTH_SHUTDOWN_META_IO_ERROR (1u << 0) +#define XFS_HEALTH_SHUTDOWN_LOG_IO_ERROR (1u << 1) +#define XFS_HEALTH_SHUTDOWN_FORCE_UMOUNT (1u << 2) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_INCORE (1u << 3) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_ONDISK (1u << 4) +#define XFS_HEALTH_SHUTDOWN_DEVICE_REMOVED (1u << 5) + +/* shutdown */ +struct xfs_health_monitor_shutdown { +/* XFS_HEALTH_SHUTDOWN_* flags */ +__u32 reasons; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1074,6 +1091,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_fs fs;struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode; +struct xfs_health_monitor_shutdown shutdown; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +4 -1
@@ -11,6 +11,9 @@ enum xfs_healthmon_type {XFS_HEALTHMON_LOST ,/* message lost */ XFS_HEALTHMON_UNMOUNT ,/* filesystem is unmounting */ +/* filesystem shutdown */ +XFS_HEALTHMON_SHUTDOWN , +/* metadata health events */ XFS_HEALTHMON_SICK ,/* runtime corruption observed */ XFS_HEALTHMON_CORRUPT, /* fsck reported corruption */
@@ -41,7 +44,7 @@ struct xfs_healthmon_event {struct {uint64_t lostcount; }; -/* mount */ +/* shutdown */ struct {unsigned int flags; };
fs/xfs/xfs_trace.h +28 -0
@@ -6010,8 +6010,32 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_read_finish);DEFINE_HEALTHMON_EVENT (xfs_healthmon_release);DEFINE_HEALTHMON_EVENT (xfs_healthmon_unmount); +TRACE_EVENT (xfs_healthmon_shutdown_hook, + TP_PROTO(const struct xfs_mount *mp, uint32_t shutdown_flags, + unsigned int events, unsigned long long lost_prev), +TP_ARGS (mp, shutdown_flags, events, lost_prev), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(uint32_t, shutdown_flags) + __field(unsigned int, events) + __field(unsigned long long, lost_prev) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->shutdown_flags = shutdown_flags; + __entry->events = events; + __entry->lost_prev = lost_prev; + ), +TP_printk ("dev %d:%d shutdown_flags %s events %u lost_prev? %llu" , +MAJOR (__entry->dev),MINOR (__entry->dev), +__print_flags (__entry->shutdown_flags,"|" ,XFS_SHUTDOWN_STRINGS ), + __entry->events, + __entry->lost_prev) +); +#define XFS_HEALTHMON_TYPE_STRINGS \ { XFS_HEALTHMON_LOST, "lost" }, \ + { XFS_HEALTHMON_SHUTDOWN, "shutdown" }, \ { XFS_HEALTHMON_UNMOUNT, "unmount" }, \ { XFS_HEALTHMON_SICK, "sick" }, \ { XFS_HEALTHMON_CORRUPT, "corrupt" }, \
@@ -6025,6 +6049,7 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_unmount); { XFS_HEALTHMON_RTGROUP, "rtgroup" }TRACE_DEFINE_ENUM (XFS_HEALTHMON_LOST ); +TRACE_DEFINE_ENUM (XFS_HEALTHMON_SHUTDOWN );TRACE_DEFINE_ENUM (XFS_HEALTHMON_UNMOUNT );TRACE_DEFINE_ENUM (XFS_HEALTHMON_SICK );TRACE_DEFINE_ENUM (XFS_HEALTHMON_CORRUPT );
@@ -6065,6 +6090,9 @@ DECLARE_EVENT_CLASS(xfs_healthmon_event_class,switch (__entry->domain) {case XFS_HEALTHMON_MOUNT :switch (__entry->type) { +case XFS_HEALTHMON_SHUTDOWN : + __entry->mask = event->flags; +break ;case XFS_HEALTHMON_LOST : __entry->lostcount = event->lostcount;break ;
fs/xfs/libxfs/xfs_healthmon.schema.json +62 -0
80 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +117 -2
218 lines not on this page · raw message
PATCH 14/19 xfs: report media errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have hooks to report media errors, connect this to the health monitor as well.
--- fs/xfs/libxfs/xfs_fs.h | 15 ++++ fs/xfs/xfs_healthmon.h | 12 ++++ fs/xfs/xfs_trace.h | 57 +++++++++++++++++ fs/xfs/libxfs/xfs_healthmon.schema.json | 65 +++++++++++++++++++ fs/xfs/xfs_healthmon.c | 106 ++++++++++++++++++++++++++++++- fs/xfs/xfs_trace.c | 1 6 files changed, 254 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +15 -0
@@ -1014,6 +1014,11 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3)#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* disk events */ +#define XFS_HEALTH_MONITOR_DOMAIN_DATADEV (5) +#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6) +#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1031,6 +1036,9 @@ struct xfs_rtgroup_geometry {/* filesystem shutdown */ #define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* media errors */ +#define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1071,6 +1079,12 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* disk media errors */ +struct xfs_health_monitor_media { +__u64 daddr; +__u64 bbcount; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1092,6 +1106,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown; +struct xfs_health_monitor_media media; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +12 -0
@@ -19,6 +19,8 @@ enum xfs_healthmon_type {XFS_HEALTHMON_CORRUPT ,/* fsck reported corruption */ XFS_HEALTHMON_HEALTHY ,/* fsck reported healthy structure */ +/* media errors */ + XFS_HEALTHMON_MEDIA_ERROR, }; enum xfs_healthmon_domain {
@@ -29,6 +31,11 @@ enum xfs_healthmon_domain {XFS_HEALTHMON_AG ,/* allocation group metadata */ XFS_HEALTHMON_INODE ,/* inode metadata */ XFS_HEALTHMON_RTGROUP ,/* realtime group metadata */ + +/* media errors */ +XFS_HEALTHMON_DATADEV , +XFS_HEALTHMON_RTDEV , + XFS_HEALTHMON_LOGDEV, }; struct xfs_healthmon_event {
@@ -66,6 +73,11 @@ struct xfs_healthmon_event {uint32_t gen;xfs_ino_t ino; }; +/* media errors */ +struct { +xfs_daddr_t daddr; +uint64_t bbcount; + }; }; };
fs/xfs/xfs_trace.h +57 -0
75 lines not on this page · raw message
fs/xfs/libxfs/xfs_healthmon.schema.json +65 -0
83 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +104 -2
212 lines not on this page · raw message
fs/xfs/xfs_trace.c +1 -0
@@ -53,6 +53,7 @@#include "xfs_zone_priv.h" #include "xfs_health.h" #include "xfs_healthmon.h" +#include "xfs_notify_failure.h" /* * We include this last to have the helpers above available for the trace
PATCH 15/19 xfs: report file io errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a file io error event hook so that we can send events about read errors, writeback errors, and directio errors to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 18 ++++ fs/xfs/xfs_healthmon.h | 16 ++++ fs/xfs/xfs_trace.h | 56 +++++++++++++ fs/xfs/libxfs/xfs_healthmon.schema.json | 77 ++++++++++++++++++ fs/xfs/xfs_healthmon.c | 131 +++++++++++++++++++++++++++++++ fs/xfs/xfs_trace.c | 1 6 files changed, 297 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +18 -0
@@ -1019,6 +1019,9 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6)#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FILERANGE (8) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1039,6 +1042,12 @@ struct xfs_rtgroup_geometry {/* media errors */ #define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_TYPE_BUFREAD (8) +#define XFS_HEALTH_MONITOR_TYPE_BUFWRITE (9) +#define XFS_HEALTH_MONITOR_TYPE_DIOREAD (10) +#define XFS_HEALTH_MONITOR_TYPE_DIOWRITE (11) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1079,6 +1088,14 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* file range events */ +struct xfs_health_monitor_filerange { +__u64 pos; +__u64 len; +__u64 ino; +__u32 gen; +}; +/* disk media errors */ struct xfs_health_monitor_media {__u64 daddr;
@@ -1107,6 +1124,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown;struct xfs_health_monitor_media media; +struct xfs_health_monitor_filerange filerange; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +16 -0
@@ -21,6 +21,12 @@ enum xfs_healthmon_type {/* media errors */ XFS_HEALTHMON_MEDIA_ERROR , + +/* file range events */ +XFS_HEALTHMON_BUFREAD , +XFS_HEALTHMON_BUFWRITE , +XFS_HEALTHMON_DIOREAD , + XFS_HEALTHMON_DIOWRITE, }; enum xfs_healthmon_domain {
@@ -36,6 +42,9 @@ enum xfs_healthmon_domain {XFS_HEALTHMON_DATADEV ,XFS_HEALTHMON_RTDEV ,XFS_HEALTHMON_LOGDEV , + +/* file range events */ + XFS_HEALTHMON_FILERANGE, }; struct xfs_healthmon_event {
@@ -78,6 +87,13 @@ struct xfs_healthmon_event {xfs_daddr_t daddr;uint64_t bbcount; }; +/* file range events */ +struct { +xfs_ino_t fino; +loff_t fpos; +uint64_t flen; +uint32_t fgen; + }; }; };
fs/xfs/xfs_trace.h +56 -0
74 lines not on this page · raw message
fs/xfs/libxfs/xfs_healthmon.schema.json +77 -0
101 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +129 -2
238 lines not on this page · raw message
fs/xfs/xfs_trace.c +1 -0
@@ -54,6 +54,7 @@#include "xfs_health.h" #include "xfs_healthmon.h" #include "xfs_notify_failure.h" +#include "xfs_file.h" /* * We include this last to have the helpers above available for the trace
PATCH 16/19 xfs: allow reconfiguration of the health monitoring device
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that we can reconfigure the health monitoring device by calling the XFS_IOC_HEALTH_MONITOR ioctl on it.
--- fs/xfs/xfs_healthmon.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
fs/xfs/xfs_healthmon.c +45 -0
@@ -23,6 +23,8 @@#include "xfs_fsops.h" #include "xfs_notify_failure.h" #include "xfs_file.h" +#include "xfs_fs.h" +#include "xfs_ioctl.h" #include <linux/anon_inodes.h> #include <linux/eventpoll.h>
@@ -1540,6 +1542,48 @@ xfs_healthmon_show_fdinfo( }#endif +/* Reconfigure the health monitor. */ +STATIC long +xfs_healthmon_reconfigure ( +struct file *file, +unsigned int cmd, +void __user *arg) +{ +struct xfs_health_monitor hmo; +struct xfs_healthmon *hm = file->private_data; + +if (copy_from_user (&hmo, arg,sizeof (hmo))) +return -EFAULT ; + +if (!xfs_healthmon_validate (&hmo)) +return -EINVAL ; + +mutex_lock (&hm->lock); + hm->format = hmo.format; + hm->verbose = !!(hmo.flags &XFS_HEALTH_MONITOR_VERBOSE ); +mutex_unlock (&hm->lock); +return 0 ; +} + +/* Handle ioctls for the health monitoring thread. */ +STATIC long +xfs_healthmon_ioctl ( +struct file *file, +unsigned int cmd, +unsigned long p) +{ +void __user *arg = (void __user *)p; + +switch (cmd) { +case XFS_IOC_HEALTH_MONITOR : +return xfs_healthmon_reconfigure (file, cmd, arg); +default : +break ; + } + +return -ENOTTY ; +} +static const struct file_operations xfs_healthmon_fops = { .owner =THIS_MODULE , #ifdef CONFIG_PROC_FS
@@ -1548,6 +1592,7 @@ static const struct file_operations xfs_healthmon_fops = { .read_iter = xfs_healthmon_read_iter, .poll = xfs_healthmon_poll, .release = xfs_healthmon_release, + .unlocked_ioctl = xfs_healthmon_ioctl, };/*
PATCH 17/19 xfs: validate fds against running healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new ioctl for the healthmon file that checks that a given fd points to the same filesystem that the healthmon file is monitoring. This allows xfs_healer to check that when it reopens a mountpoint to perform repairs, the file that it gets matches the filesystem that generated the corruption report.
(Note that xfs_healer doesn't maintain an open fd to a filesystem that it's monitoring so that it doesn't pin the mount.)
--- fs/xfs/libxfs/xfs_fs.h | 10 ++++++++++ fs/xfs/xfs_healthmon.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+)
fs/xfs/libxfs/xfs_fs.h +10 -0
@@ -1149,6 +1149,15 @@ struct xfs_health_monitor {/* Return events in JSON format */ #define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* +* Check that a given fd points to the same filesystem that the health monitor +* is monitoring. +*/ +struct xfs_health_samefs { +__s32 fd; +__u32 flags;/* zero for now */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1189,6 +1198,7 @@ struct xfs_health_monitor {#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor) +#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.c +32 -0
@@ -1565,6 +1565,36 @@ xfs_healthmon_reconfigure(return 0 ; } +/* Does the fd point to the same filesystem as the one we're monitoring? */ +STATIC long +xfs_healthmon_samefs ( +struct file *file, +unsigned int cmd, +void __user *arg) +{ +struct xfs_health_samefs hms; +struct xfs_healthmon *hm = file->private_data; +struct inode *hms_inode; +int ret =0 ; + +if (copy_from_user (&hms, arg,sizeof (hms))) +return -EFAULT ; + +if (hms.flags) +return -EINVAL ; + +CLASS (fd, hms_fd)(hms.fd); +if (fd_empty (hms_fd)) +return -EBADF ; + + hms_inode =file_inode (fd_file (hms_fd)); +mutex_lock (&hm->lock); +if (!hm->mp || hm->mp->m_super != hms_inode->i_sb) + ret = -ESTALE ; +mutex_unlock (&hm->lock); +return ret; +} +/* Handle ioctls for the health monitoring thread. */ STATIC long xfs_healthmon_ioctl(
@@ -1577,6 +1607,8 @@ xfs_healthmon_ioctl(switch (cmd) {case XFS_IOC_HEALTH_MONITOR :return xfs_healthmon_reconfigure (file, cmd, arg); +case XFS_IOC_HEALTH_SAMEFS : +return xfs_healthmon_samefs (file, cmd, arg);default :break ; }
PATCH 18/19 xfs: add media error reporting ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new privileged ioctl so that xfs_scrub can report media errors to the kernel for further processing.
--- fs/xfs/libxfs/xfs_fs.h | 16 +++++++++++++ fs/xfs/xfs_notify_failure.h | 8 ++++++ fs/xfs/xfs_trace.h | 2 -- fs/xfs/Makefile | 6 +---- fs/xfs/xfs_healthmon.c | 2 -- fs/xfs/xfs_ioctl.c | 3 ++ fs/xfs/xfs_notify_failure.c | 53 ++++++++++++++++++++++++++++++++++++++++++- 7 files changed, 79 insertions(+), 11 deletions(-)
fs/xfs/libxfs/xfs_fs.h +16 -0
@@ -1158,6 +1158,21 @@ struct xfs_health_samefs {__u32 flags;/* zero for now */ }; +/* Report a media error */ +struct xfs_media_error { +__u64 flags;/* flags */ +__u64 daddr;/* disk address of range */ +__u64 bbcount;/* length, in 512b blocks */ +__u64 pad;/* zero */ +}; + +#define XFS_MEDIA_ERROR_DATADEV (1)/* data device */ +#define XFS_MEDIA_ERROR_LOGDEV (2)/* external log device */ +#define XFS_MEDIA_ERROR_RTDEV (3)/* realtime device */ + +/* bottom byte of flags is the device code */ +#define XFS_MEDIA_ERROR_DEVMASK (0xFF) +/* * ioctl commands that are used by Linux filesystems */
@@ -1199,6 +1214,7 @@ struct xfs_health_samefs {#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs) +#define XFS_IOC_MEDIA_ERROR _IOW ('X', 70, struct xfs_media_error)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_notify_failure.h +7 -1
@@ -6,7 +6,9 @@#ifndef __XFS_NOTIFY_FAILURE_H__#define __XFS_NOTIFY_FAILURE_H__ +#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX )extern const struct dax_holder_operations xfs_dax_holder_operations; +#endif enum xfs_failed_device { XFS_FAILED_DATADEV,
@@ -14,7 +16,7 @@ enum xfs_failed_device { XFS_FAILED_RTDEV, }; -#if defined(CONFIG_XFS_LIVE_HOOKS) && defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX) +#if defined(CONFIG_XFS_LIVE_HOOKS) struct xfs_media_error_params { struct xfs_mount *mp; enum xfs_failed_device fdev;
@@ -46,4 +48,8 @@ struct xfs_media_error_hook { };# define xfs_media_error_hook_setup (...) ((void)0)#endif /* CONFIG_XFS_LIVE_HOOKS */ +struct xfs_media_error ; +int xfs_ioc_media_error (struct xfs_mount *mp, +struct xfs_media_error __user *arg); +#endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_trace.h +0 -2
@@ -6214,7 +6214,6 @@ TRACE_EVENT(xfs_healthmon_metadata_hook, __entry->lost_prev) ); -#if defined(CONFIG_XFS_LIVE_HOOKS ) && defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX ) TRACE_EVENT(xfs_healthmon_media_error_hook, TP_PROTO(const struct xfs_media_error_params *p, unsigned int events, unsigned long long lost_prev),
@@ -6262,7 +6261,6 @@ TRACE_EVENT(xfs_healthmon_media_error_hook, __entry->events, __entry->lost_prev) ); -#endif #define XFS_FILE_IOERROR_STRINGS \ { XFS_FILE_IOERROR_BUFFERED_READ, "readahead" }, \
fs/xfs/Makefile +1 -5
@@ -98,6 +98,7 @@ xfs-y += xfs_aops.o \ xfs_message.o \ xfs_mount.o \ xfs_mru_cache.o \ + xfs_notify_failure.o \ xfs_pwork.o \ xfs_reflink.o \ xfs_stats.o \
@@ -148,11 +149,6 @@ xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o xfs-$(CONFIG_EXPORTFS_BLOCK_OPS) += xfs_pnfs.o -# notify failure -ifeq ($(CONFIG_MEMORY_FAILURE),y) -xfs-$(CONFIG_FS_DAX) += xfs_notify_failure.o -endif - xfs-$(CONFIG_XFS_DRAIN_INTENTS) += xfs_drain.o xfs-$(CONFIG_XFS_LIVE_HOOKS) += xfs_hooks.o xfs-$(CONFIG_XFS_MEMORY_BUFS) += xfs_buf_mem.o
fs/xfs/xfs_healthmon.c +0 -2
@@ -527,7 +527,6 @@ xfs_healthmon_shutdown_hook(return NOTIFY_DONE ; } -#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX )/* Add a media error event to the reporting queue. */ STATIC int xfs_healthmon_media_error_hook(
@@ -578,7 +577,6 @@ xfs_healthmon_media_error_hook(mutex_unlock (&hm->lock);return NOTIFY_DONE ; } -#endif /* Add a file io error event to the reporting queue. */ STATIC int
fs/xfs/xfs_ioctl.c +3 -0
@@ -42,6 +42,7 @@#include "xfs_handle.h" #include "xfs_rtgroup.h" #include "xfs_healthmon.h" +#include "xfs_notify_failure.h" #include <linux/mount.h> #include <linux/fileattr.h>
@@ -1424,6 +1425,8 @@ xfs_file_ioctl(case XFS_IOC_HEALTH_MONITOR :return xfs_ioc_health_monitor (mp, arg); +case XFS_IOC_MEDIA_ERROR : +return xfs_ioc_media_error (mp, arg);default :return -ENOTTY ;
fs/xfs/xfs_notify_failure.c +52 -1
64 lines not on this page · raw message
PATCH 19/19 xfs: send uevents when major filesystem events happen
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Send uevents when we mount, unmount, and shut down the filesystem, so that we can trigger systemd services when major events happen.
--- fs/xfs/xfs_super.h | 13 +++++++ fs/xfs/xfs_fsops.c | 18 ++++++++++ fs/xfs/xfs_super.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 125 insertions(+)
fs/xfs/xfs_super.h +13 -0
@@ -101,4 +101,17 @@ extern struct workqueue_struct *xfs_discard_wq;struct dentry *xfs_debugfs_mkdir (const char *name,struct dentry *parent); +#define XFS_UEVENT_BUFLEN ( \ + sizeof("SID=") + sizeof_field(struct super_block, s_id) + \ + sizeof("UUID=") + UUID_STRING_LEN + \ + sizeof("META_UUID=") + UUID_STRING_LEN) + +#define XFS_UEVENT_STR_PTRS \ + NULL,/* sid */ \ +NULL ,/* uuid */ \ + NULL /* metauuid */ + +int xfs_format_uevent_strings (struct xfs_mount *mp,char *buf,ssize_t buflen, +char **env); +#endif /* __XFS_SUPER_H__ */
fs/xfs/xfs_fsops.c +18 -0
@@ -537,6 +537,23 @@ xfs_shutdown_hook_setup(# define xfs_shutdown_hook (...) ((void)0)#endif /* CONFIG_XFS_LIVE_HOOKS */ +static void +xfs_send_shutdown_uevent ( +struct xfs_mount *mp) +{ +char buf[XFS_UEVENT_BUFLEN ]; +char *env[] = { +"TYPE=shutdown" , +XFS_UEVENT_STR_PTRS , +NULL , + }; +int error; + + error =xfs_format_uevent_strings (mp, buf,sizeof (buf), &env[2 ]); +if (!error) +kobject_uevent_env (&mp->m_kobj.kobject,KOBJ_OFFLINE , env); +} +/* * Force a shutdown of the filesystem instantly while keeping the filesystem * consistent. We don't do an unmount here; just shutdown the shop, make sure
@@ -587,6 +604,7 @@ xfs_do_force_shutdown( }trace_xfs_force_shutdown (mp, tag, flags, fname, lnnum); +xfs_send_shutdown_uevent (mp); xfs_alert_tag(mp, tag, "%s (0x%x) detected at %pS (%s:%d). Shutting down filesystem.",
fs/xfs/xfs_super.c +94 -0
124 lines not on this page · raw message
PATCH v2 1/2 xfsprogs: autonomous self healing of filesystems
In reply to Darrick J. WongHi all,
This patchset builds new functionality to deliver live information about filesystem health events to userspace. This is done by creating an anonymous file that can be read() for events by userspace programs. Events are captured by hooking various parts of XFS and iomap so that metadata health failures, file I/O errors, and major changes in filesystem state (unmounts, shutdowns, etc.) can be observed by programs.
When an event occurs, the hook functions queue an event object to each event anonfd for later processing. Programs must have CAP_SYS_ADMIN to open the anonfd and there's a maximum event lag to prevent resource overconsumption. The events themselves can be read() from the anonfd either as json objects for human readability, or as C structs for daemons.
In userspace, we create a new daemon program that will read the event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some udev rules.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* xfs: create hooks for monitoring health updates
* xfs: create a special file to pass filesystem health to userspace
* xfs: create event queuing, formatting, and discovery infrastructure
* xfs: report metadata health events through healthmon
* xfs: report shutdown events through healthmon
* xfs: report media errors through healthmon
* xfs: report file io errors through healthmon
* xfs: validate fds against running healthmon
* xfs: add media error reporting ioctl
* xfs_io: monitor filesystem health events
* xfs_io: add a media error reporting command
* xfs_healer: create daemon to listen for health events
* xfs_healer: check events against schema
* xfs_healer: enable repairing filesystems
* xfs_healer: check for fs features needed for effective repairs
* xfs_healer: use getparents to look up file names
* builddefs: refactor udev directory specification
* xfs_healer: create a background monitoring service
* xfs_healer: don't start service if kernel support unavailable
* xfs_healer: use the autofsck fsproperty to select mode
* xfs_healer: run full scrub after lost corruption events or targeted repair failure
* xfs_healer: use getmntent to find moved filesystems
* xfs_healer: validate that repair fds point to the monitored fs
* xfs_healer: add a manual page
* xfs_scrub: report media scrub failures to the kernel
* debian: enable xfs_healer on the root filesystem by default
--- io/io.h | 1 libxfs/xfs_fs.h | 173 +++++ libxfs/xfs_health.h | 52 + Makefile | 5 configure.ac | 8 debian/control | 2 debian/postinst | 8 debian/prerm | 13 debian/rules | 2 healer/Makefile | 68 ++ healer/system-xfs_healer.slice | 31 + healer/xfs_healer.py.in | 1432 ++++++++++++++++++++++++++++++++++++++ healer/xfs_healer.rules | 7 healer/[email protected] | 108 +++ healer/xfs_healer_start | 17 include/builddefs.in | 5 io/Makefile | 1 io/healthmon.c | 183 +++++ io/init.c | 1 io/shutdown.c | 113 +++ libxfs/Makefile | 10 libxfs/xfs_healthmon.schema.json | 648 +++++++++++++++++ m4/package_services.m4 | 30 - man/man8/xfs_healer.8 | 85 ++ man/man8/xfs_io.8 | 46 + scrub/Makefile | 7 scrub/phase6.c | 25 + 27 files changed, 3054 insertions(+), 27 deletions(-) create mode 100644 debian/prerm create mode 100644 healer/Makefile create mode 100644 healer/system-xfs_healer.slice create mode 100644 healer/xfs_healer.py.in create mode 100644 healer/xfs_healer.rules create mode 100644 healer/[email protected] create mode 100755 healer/xfs_healer_start create mode 100644 io/healthmon.c create mode 100644 libxfs/xfs_healthmon.schema.json create mode 100644 man/man8/xfs_healer.8
PATCH 1/26 xfs: create hooks for monitoring health updates
From: Darrick J. Wong <[email protected]>
Create hooks for monitoring health events.
--- libxfs/xfs_health.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
libxfs/xfs_health.h +47 -0
@@ -289,4 +289,51 @@ void xfs_bulkstat_health(struct xfs_inode *ip, struct xfs_bulkstat *bs);#define xfs_metadata_is_sick (error) \ (unlikely((error) == -EFSCORRUPTED || (error) == -EFSBADCRC)) +/* +* Parameters for tracking health updates. The enum below is passed as the +* hook function argument. +*/ +enum xfs_health_update_type { +XFS_HEALTHUP_SICK =1 ,/* runtime corruption observed */ +XFS_HEALTHUP_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHUP_HEALTHY ,/* fsck reported healthy structure */ +XFS_HEALTHUP_UNMOUNT ,/* filesystem is unmounting */ +}; + +/* Where in the filesystem was the event observed? */ +enum xfs_health_update_domain { +XFS_HEALTHUP_FS =1 ,/* main filesystem */ +XFS_HEALTHUP_AG ,/* allocation group */ +XFS_HEALTHUP_INODE ,/* inode */ +XFS_HEALTHUP_RTGROUP ,/* realtime group */ +}; + +struct xfs_health_update_params { +/* XFS_HEALTHUP_INODE */ +xfs_ino_t ino; +uint32_t gen; + +/* XFS_HEALTHUP_AG/RTGROUP */ +uint32_t group; + +/* XFS_SICK_* flags */ +unsigned int old_mask; +unsigned int new_mask; + +enum xfs_health_update_domain domain; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_health_hook { +struct xfs_hook health_hook; +}; + +void xfs_health_hook_disable (void ); +void xfs_health_hook_enable (void ); + +int xfs_health_hook_add (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_del (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_HEALTH_H__ */
PATCH 2/26 xfs: create a special file to pass filesystem health to userspace
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an ioctl that installs a file descriptor backed by an anon_inode file that will convey filesystem health events to userspace.
--- libxfs/xfs_fs.h | 8 ++++++++ 1 file changed, 8 insertions(+)
libxfs/xfs_fs.h +8 -0
@@ -1003,6 +1003,13 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +struct xfs_health_monitor { +__u64 flags;/* flags */ +__u8 format;/* output format */ +__u8 pad1[7 ];/* zeroes */ +__u64 pad2[2 ];/* zeroes */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1042,6 +1049,7 @@ struct xfs_rtgroup_geometry {#define XFS_IOC_GETPARENTS_BY_HANDLE _IOWR('X', 63, struct xfs_getparents_by_handle)#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry) +#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 3/26 xfs: create event queuing, formatting, and discovery infrastructure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle.
Here, we've chosen json to export information to userspace. The structured key-value nature of json gives us enormous flexibility to modify the schema of what we'll send to userspace because we can add new keys at any time. Userspace can use whatever json parsers are available to consume the events and will not be confused by keys they don't recognize.
Note that we do NOT allow sending json back to the kernel, nor is there any intent to do that.
--- libxfs/xfs_fs.h | 50 +++++++++++++++ libxfs/xfs_healthmon.schema.json | 129 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 179 insertions(+) create mode 100644 libxfs/xfs_healthmon.schema.json
libxfs/xfs_fs.h +50 -0
@@ -1003,6 +1003,45 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +/* Health monitor event domains */ + +/* affects the whole fs */ +#define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) + +/* Health monitor event types */ + +/* status of the monitor itself */ +#define XFS_HEALTH_MONITOR_TYPE_RUNNING (0) +#define XFS_HEALTH_MONITOR_TYPE_LOST (1) + +/* lost events */ +struct xfs_health_monitor_lost { +__u64 count; +}; + +struct xfs_health_monitor_event { +/* XFS_HEALTH_MONITOR_DOMAIN_* */ +__u32 domain; + +/* XFS_HEALTH_MONITOR_TYPE_* */ +__u32 type; + +/* Timestamp of the event, in nanoseconds since the Unix epoch */ +__u64 time_ns; + +/* +* Details of the event. The primary clients are written in python +* and rust, so break this up because bindgen hates anonymous structs +* and unions. +*/ +union { +struct xfs_health_monitor_lost lost; + } e; + +/* zeroes */ +__u64 pad[2 ]; +}; +struct xfs_health_monitor {__u64 flags;/* flags */ __u8 format;/* output format */
@@ -1010,6 +1049,17 @@ struct xfs_health_monitor {__u64 pad2[2 ];/* zeroes */ }; +/* Return all health status events, not just deltas */ +#define XFS_HEALTH_MONITOR_VERBOSE (1ULL << 0) + +#define XFS_HEALTH_MONITOR_ALL (XFS_HEALTH_MONITOR_VERBOSE) + +/* Return events in a C structure */ +#define XFS_HEALTH_MONITOR_FMT_CSTRUCT (0) + +/* Return events in JSON format */ +#define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* * ioctl commands that are used by Linux filesystems */
libxfs/xfs_healthmon.schema.json new file +129 -0
129 lines not on this page · raw message
PATCH 4/26 xfs: report metadata health events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a metadata health event hook so that we can send events to userspace as we collect information. The unmount hook severs the weak reference between the health monitor and the filesystem it's monitoring; when this happens, we stop reporting events because there's no longer any point.
--- libxfs/xfs_fs.h | 38 +++++ libxfs/xfs_health.h | 5 + libxfs/xfs_healthmon.schema.json | 315 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 358 insertions(+)
libxfs/xfs_fs.h +38 -0
@@ -1008,17 +1008,52 @@ struct xfs_rtgroup_geometry {/* affects the whole fs */ #define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FS (1) +#define XFS_HEALTH_MONITOR_DOMAIN_AG (2) +#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3) +#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* Health monitor event types */ /* status of the monitor itself */ #define XFS_HEALTH_MONITOR_TYPE_RUNNING (0)#define XFS_HEALTH_MONITOR_TYPE_LOST (1) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_TYPE_SICK (2) +#define XFS_HEALTH_MONITOR_TYPE_CORRUPT (3) +#define XFS_HEALTH_MONITOR_TYPE_HEALTHY (4) + +/* filesystem was unmounted */ +#define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* lost events */ struct xfs_health_monitor_lost {__u64 count; }; +/* fs/rt metadata */ +struct xfs_health_monitor_fs { +/* XFS_FSOP_GEOM_SICK_* flags */ +__u32 mask; +}; + +/* ag/rtgroup metadata */ +struct xfs_health_monitor_group { +/* XFS_{AG,RTGROUP}_SICK_* flags */ +__u32 mask; +__u32 gno; +}; + +/* inode metadata */ +struct xfs_health_monitor_inode { +/* XFS_BS_SICK_* flags */ +__u32 mask; +__u32 gen; +__u64 ino; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1036,6 +1071,9 @@ struct xfs_health_monitor_event {*/ union {struct xfs_health_monitor_lost lost; +struct xfs_health_monitor_fs fs; +struct xfs_health_monitor_group group; +struct xfs_health_monitor_inode inode; } e;/* zeroes */
libxfs/xfs_health.h +5 -0
@@ -336,4 +336,9 @@ void xfs_health_hook_del(struct xfs_mount *mp, struct xfs_health_hook *hook);void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn);#endif /* CONFIG_XFS_LIVE_HOOKS */ +unsigned int xfs_healthmon_inode_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_rtgroup_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_perag_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_fs_mask (unsigned int sick_mask); +#endif /* __XFS_HEALTH_H__ */
libxfs/xfs_healthmon.schema.json +315 -0
333 lines not on this page · raw message
PATCH 5/26 xfs: report shutdown events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a shutdown hook so that we can send notifications to userspace.
--- libxfs/xfs_fs.h | 18 +++++++++++ libxfs/xfs_healthmon.schema.json | 62 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+)
libxfs/xfs_fs.h +18 -0
@@ -1028,6 +1028,9 @@ struct xfs_rtgroup_geometry {/* filesystem was unmounted */ #define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* filesystem shutdown */ +#define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1054,6 +1057,20 @@ struct xfs_health_monitor_inode {__u64 ino; }; +/* shutdown reasons */ +#define XFS_HEALTH_SHUTDOWN_META_IO_ERROR (1u << 0) +#define XFS_HEALTH_SHUTDOWN_LOG_IO_ERROR (1u << 1) +#define XFS_HEALTH_SHUTDOWN_FORCE_UMOUNT (1u << 2) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_INCORE (1u << 3) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_ONDISK (1u << 4) +#define XFS_HEALTH_SHUTDOWN_DEVICE_REMOVED (1u << 5) + +/* shutdown */ +struct xfs_health_monitor_shutdown { +/* XFS_HEALTH_SHUTDOWN_* flags */ +__u32 reasons; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1074,6 +1091,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_fs fs;struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode; +struct xfs_health_monitor_shutdown shutdown; } e;/* zeroes */
libxfs/xfs_healthmon.schema.json +62 -0
80 lines not on this page · raw message
PATCH 6/26 xfs: report media errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have hooks to report media errors, connect this to the health monitor as well.
--- libxfs/xfs_fs.h | 15 +++++++++ libxfs/xfs_healthmon.schema.json | 65 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+)
libxfs/xfs_fs.h +15 -0
@@ -1014,6 +1014,11 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3)#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* disk events */ +#define XFS_HEALTH_MONITOR_DOMAIN_DATADEV (5) +#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6) +#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1031,6 +1036,9 @@ struct xfs_rtgroup_geometry {/* filesystem shutdown */ #define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* media errors */ +#define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1071,6 +1079,12 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* disk media errors */ +struct xfs_health_monitor_media { +__u64 daddr; +__u64 bbcount; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1092,6 +1106,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown; +struct xfs_health_monitor_media media; } e;/* zeroes */
libxfs/xfs_healthmon.schema.json +65 -0
83 lines not on this page · raw message
PATCH 7/26 xfs: report file io errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a file io error event hook so that we can send events about read errors, writeback errors, and directio errors to userspace.
--- libxfs/xfs_fs.h | 18 +++++++++ libxfs/xfs_healthmon.schema.json | 77 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+)
libxfs/xfs_fs.h +18 -0
@@ -1019,6 +1019,9 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6)#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FILERANGE (8) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1039,6 +1042,12 @@ struct xfs_rtgroup_geometry {/* media errors */ #define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_TYPE_BUFREAD (8) +#define XFS_HEALTH_MONITOR_TYPE_BUFWRITE (9) +#define XFS_HEALTH_MONITOR_TYPE_DIOREAD (10) +#define XFS_HEALTH_MONITOR_TYPE_DIOWRITE (11) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1079,6 +1088,14 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* file range events */ +struct xfs_health_monitor_filerange { +__u64 pos; +__u64 len; +__u64 ino; +__u32 gen; +}; +/* disk media errors */ struct xfs_health_monitor_media {__u64 daddr;
@@ -1107,6 +1124,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown;struct xfs_health_monitor_media media; +struct xfs_health_monitor_filerange filerange; } e;/* zeroes */
libxfs/xfs_healthmon.schema.json +77 -0
101 lines not on this page · raw message
PATCH 8/26 xfs: validate fds against running healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new ioctl for the healthmon file that checks that a given fd points to the same filesystem that the healthmon file is monitoring. This allows xfs_healer to check that when it reopens a mountpoint to perform repairs, the file that it gets matches the filesystem that generated the corruption report.
(Note that xfs_healer doesn't maintain an open fd to a filesystem that it's monitoring so that it doesn't pin the mount.)
--- libxfs/xfs_fs.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
libxfs/xfs_fs.h +10 -0
@@ -1149,6 +1149,15 @@ struct xfs_health_monitor {/* Return events in JSON format */ #define XFS_HEALTH_MONITOR_FMT_JSON (1) +/* +* Check that a given fd points to the same filesystem that the health monitor +* is monitoring. +*/ +struct xfs_health_samefs { +__s32 fd; +__u32 flags;/* zero for now */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1189,6 +1198,7 @@ struct xfs_health_monitor {#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor) +#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 9/26 xfs: add media error reporting ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new privileged ioctl so that xfs_scrub can report media errors to the kernel for further processing.
--- libxfs/xfs_fs.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
libxfs/xfs_fs.h +16 -0
@@ -1158,6 +1158,21 @@ struct xfs_health_samefs {__u32 flags;/* zero for now */ }; +/* Report a media error */ +struct xfs_media_error { +__u64 flags;/* flags */ +__u64 daddr;/* disk address of range */ +__u64 bbcount;/* length, in 512b blocks */ +__u64 pad;/* zero */ +}; + +#define XFS_MEDIA_ERROR_DATADEV (1)/* data device */ +#define XFS_MEDIA_ERROR_LOGDEV (2)/* external log device */ +#define XFS_MEDIA_ERROR_RTDEV (3)/* realtime device */ + +/* bottom byte of flags is the device code */ +#define XFS_MEDIA_ERROR_DEVMASK (0xFF) +/* * ioctl commands that are used by Linux filesystems */
@@ -1199,6 +1214,7 @@ struct xfs_health_samefs {#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs) +#define XFS_IOC_MEDIA_ERROR _IOW ('X', 70, struct xfs_media_error)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 10/26 xfs_io: monitor filesystem health events
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a subcommand to monitor for health events generated by the kernel.
--- io/io.h | 1 io/Makefile | 1 io/healthmon.c | 183 +++++++++++++++++++++++++++++++++++++++++++++++++++++ io/init.c | 1 man/man8/xfs_io.8 | 25 +++++++ 5 files changed, 211 insertions(+) create mode 100644 io/healthmon.c
io/io.h +1 -0
@@ -162,3 +162,4 @@ extern void bulkstat_init(void);void exchangerange_init (void );void fsprops_init (void );void aginfo_init (void ); +void healthmon_init (void );
io/Makefile +1 -0
@@ -25,6 +25,7 @@ CFILES = \ fsuuid.c \ fsync.c \ getrusage.c \ + healthmon.c \ imap.c \ init.c \ inject.c \
io/healthmon.c new file +183 -0
183 lines not on this page · raw message
io/init.c +1 -0
@@ -92,6 +92,7 @@ init_commands(void)crc32cselftest_init ();exchangerange_init ();fsprops_init (); +healthmon_init (); }/*
man/man8/xfs_io.8 +25 -0
@@ -1356,6 +1356,31 @@ .SH FILESYSTEM COMMANDS .B thaw Undo the effects of a filesystem freeze operation. Only available in expert mode and requires privileges. +.TP +.BI "healthmon [ \-c " bufsize " ] [ \-c ] [ \-d " delay_ms " ] [ \-p ] [ \-v ]" +Watch for filesystem health events and write them to the console. +.RE +.RS 1.0i +.PD 0 +.TP +.BI "\-b " bufsize +Use a buffer of this size to read events from the kernel. +.TP +.BI \-c +Close the open file and replace it with the monitor file. +.TP +.BI "\-d " delay_ms +Sleep for this long between read attempts. +.TP +.B \-p +Probe for the existence of the functionality by opening the monitoring fd and +closing it immediately. +.TP +.BI \-v +Request all health events, even if nothing changed. +.PD +.RE + .TP .BI "inject [ " tag " ]" Inject errors into a filesystem to observe filesystem behavior at
PATCH 11/26 xfs_io: add a media error reporting command
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a subcommand to invoke the media error ioctl to make sure it works.
--- io/shutdown.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++ man/man8/xfs_io.8 | 21 ++++++++++ 2 files changed, 133 insertions(+), 1 deletion(-)
io/shutdown.c +112 -1
@@ -53,6 +53,115 @@ shutdown_help(void) "\n")); } +static void +mediaerror_help (void ) +{ +printf (_ ( +"\n" +" Report a media error on the data device to the filesystem.\n" +"\n" +" -l -- Report against the log device.\n" +" -r -- Report against the realtime device.\n" +"\n" +" offset is the byte offset of the start of the failed range. If offset is\n" +" specified, mapping length may (optionally) be specified as well." +"\n" +" length is the byte length of the failed range.\n" +"\n" +" If neither offset nor length are specified, the media error report will\n" +" be made against the entire device." +"\n" )); +} + +static int +mediaerror_f ( +int argc, +char **argv) +{ +struct xfs_media_error me = { + .daddr =0 , + .bbcount =-1ULL , + .flags =XFS_MEDIA_ERROR_DATADEV , + }; +long long l; +size_t fsblocksize, fssectsize; +int c, ret; + +init_cvtnum (&fsblocksize, &fssectsize); + +while ((c =getopt (argc, argv,"lr" )) !=EOF ) { +switch (c) { +case 'l' : + me.flags = (me.flags & ~XFS_MEDIA_ERROR_DEVMASK ) | +XFS_MEDIA_ERROR_LOGDEV ; +break ; +case 'r' : + me.flags = (me.flags & ~XFS_MEDIA_ERROR_DEVMASK ) | +XFS_MEDIA_ERROR_RTDEV ; +break ; +default : +mediaerror_help (); + exitcode =1 ; +return 0 ; + } + } + +/* Range start (optional) */ +if (optind < argc) { + l =cvtnum (fsblocksize, fssectsize, argv[optind]); +if (l <0 ) { +printf ("non-numeric offset argument -- %s\n" , + argv[optind]); + exitcode =1 ; +return 0 ; + } + + me.daddr = l /512 ; + optind++; + } + +/* Range length (optional if range start was specified) */ +if (optind < argc) { + l =cvtnum (fsblocksize, fssectsize, argv[optind]); +if (l <0 ) { +printf ("non-numeric len argument -- %s\n" , + argv[optind]); + exitcode =1 ; +return 0 ; + } + + me.bbcount =howmany (l,512 ); + optind++; + } + +if (optind < argc) { +printf ("too many arguments -- %s\n" , argv[optind]); + exitcode =1 ; +return 0 ; + } + + ret =ioctl (file->fd,XFS_IOC_MEDIA_ERROR , &me); +if (ret) { +fprintf (stderr, +"%s: ioctl(XFS_IOC_MEDIA_ERROR) [\"%s\"]: %s\n" , + progname, file->name,strerror (errno)); + exitcode =1 ; +return 0 ; + } + +return 0 ; +} + +static struct cmdinfo mediaerror_cmd = { + .name ="mediaerror" , + .cfunc = mediaerror_f, + .argmin =0 , + .argmax =-1 , + .flags =CMD_FLAG_ONESHOT |CMD_NOMAP_OK , + .args ="[-lr] [offset [length]]" , + .help = mediaerror_help, +}; +void shutdown_init (void ) {
@@ -66,6 +175,8 @@ shutdown_init(void) shutdown_cmd.oneline =_ ("shuts down the filesystem where the current file resides" ); -if (expert) +if (expert) {add_command (&shutdown_cmd); +add_command (&mediaerror_cmd); + } }
man/man8/xfs_io.8 +21 -0
@@ -1389,6 +1389,27 @@ .SH FILESYSTEM COMMANDS argument, displays the list of error tags available. Only available in expert mode and requires privileges. +.TP +.BI "mediaerror [ \-lr ] [ " offset " [ " length " ]]" +Report a media error against the data device of an XFS filesystem. +The +.I offset +and +.I length +parameters are specified in units of bytes. +If neither are specified, the entire device will be reported. +.RE +.RS 1.0i +.PD 0 +.TP +.BI \-l +Report against the log device instead of the data device. +.TP +.BI \-r +Report against the realtime device instead of the data device. +.PD +.RE + .TP .BI "rginfo [ \-r " rgno " ]" Show information about or update the state of realtime allocation groups.
PATCH 12/26 xfs_healer: create daemon to listen for health events
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a daemon program that can listen for and log health events. Eventually this will be used to self-heal filesystems in real time.
--- Makefile | 5 + configure.ac | 6 + healer/Makefile | 34 ++++ healer/xfs_healer.py.in | 368 +++++++++++++++++++++++++++++++++++++++++++++++ include/builddefs.in | 1 5 files changed, 414 insertions(+) create mode 100644 healer/Makefile create mode 100644 healer/xfs_healer.py.in
Makefile +5 -0
@@ -69,6 +69,10 @@ ifeq ("$(ENABLE_SCRUB)","yes") TOOL_SUBDIRS += scrub endif +ifeq ("$(ENABLE_SCRUB)","yes") +TOOL_SUBDIRS += healer +endif + ifneq ("$(XGETTEXT)","") TOOL_SUBDIRS += po endif
@@ -100,6 +104,7 @@ mkfs: libxcmd spaceman: libxcmd libhandle scrub: libhandle libxcmd rtcp: libfrog +healer: ifeq ($(HAVE_BUILDDEFS), yes) include $(BUILDRULES)
configure.ac +6 -0
@@ -110,6 +110,12 @@ AC_ARG_ENABLE(libicu, [ --enable-libicu=[yes/no] Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],, enable_libicu=probe) +# Enable xfs_healer build +AC_ARG_ENABLE(healer, +[ --enable-healer=[yes/no] Enable build of xfs_healer utility [[default=yes]]],, + enable_healer=yes) +AC_SUBST(enable_healer) + # # If the user specified a libdir ending in lib64 do not append another # 64 to the library names.
healer/Makefile new file +34 -0
@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2024-2025 Oracle. All Rights Reserved. +# + +TOPDIR = .. +builddefs=$(TOPDIR)/include/builddefs +include $(builddefs) + +XFS_HEALER_PROG = xfs_healer.py +INSTALL_HEALER = install-healer + +LDIRT = $(XFS_HEALER_PROG) + +default: $(XFS_HEALER_PROG) + +$(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext.py + @echo " [SED] $@" + $(Q)$(SED) -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e '/@INIT_GETTEXT@/r $(TOPDIR)/libfrog/gettext.py' \ + -e '/@INIT_GETTEXT@/d' \ + < $< > $@ + $(Q)chmod a+x $@ + +include $(BUILDRULES) + +install: $(INSTALL_HEALER) + +install-healer: default + $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) + $(INSTALL) -m 755 $(XFS_HEALER_PROG) $(PKG_LIBEXEC_DIR)/xfs_healer + +install-dev: + +-include .dep
healer/xfs_healer.py.in new file +368 -0
368 lines not on this page · raw message
include/builddefs.in +1 -0
@@ -91,6 +91,7 @@ ENABLE_SHARED = @enable_shared@ ENABLE_GETTEXT = @enable_gettext@ ENABLE_EDITLINE = @enable_editline@ ENABLE_SCRUB = @enable_scrub@ +ENABLE_HEALER = @enable_healer@ HAVE_ZIPPED_MANPAGES = @have_zipped_manpages@
PATCH 13/26 xfs_healer: check events against schema
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Validate that the event objects that we get from the kernel actually obey the schema that the kernel publishes.
--- healer/Makefile | 1 + healer/xfs_healer.py.in | 62 +++++++++++++++++++++++++++++++++++++++++++++++ libxfs/Makefile | 10 +++++--- 3 files changed, 70 insertions(+), 3 deletions(-)
healer/Makefile +1 -0
@@ -18,6 +18,7 @@ $(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext $(Q)$(SED) -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e '/@INIT_GETTEXT@/r $(TOPDIR)/libfrog/gettext.py' \ -e '/@INIT_GETTEXT@/d' \ + -e "s|@pkg_data_dir@|$(PKG_DATA_DIR)|g" \ < $< > $@ $(Q)chmod a+x $@
healer/xfs_healer.py.in +62 -0
92 lines not on this page · raw message
libxfs/Makefile +7 -3
@@ -151,6 +151,8 @@ EXTRA_OBJECTS=\ LDIRT += $(EXTRA_OBJECTS) +JSON_SCHEMAS=xfs_healthmon.schema.json + # # Tracing flags: # -DMEM_DEBUG all zone memory use
@@ -174,7 +176,7 @@ LTLIBS = $(LIBPTHREAD) $(LIBRT) # don't try linking xfs_repair with a debug libxfs. DEBUG = -DNDEBUG -default: ltdepend $(LTLIBRARY) $(EXTRA_OBJECTS) +default: ltdepend $(LTLIBRARY) $(EXTRA_OBJECTS) $(JSON_SCHEMAS) %dummy.o: %dummy.cpp @echo " [CXXD] $@"
@@ -196,14 +198,16 @@ MAKECXXDEP := $(MAKEDEPEND) $(CXXFLAGS) include $(BUILDRULES) install: default - $(INSTALL) -m 755 -d $(PKG_INC_DIR) + $(INSTALL) -m 755 -d $(PKG_DATA_DIR) + $(INSTALL) -m 644 $(JSON_SCHEMAS) $(PKG_DATA_DIR) install-headers: $(addsuffix -hdrs, $(PKGHFILES)) %-hdrs: $(Q)$(LN_S) -f $(CURDIR)/$* $(TOPDIR)/include/xfs/$* -install-dev: install +install-dev: default + $(INSTALL) -m 755 -d $(PKG_INC_DIR) $(INSTALL) -m 644 $(PKGHFILES) $(PKG_INC_DIR) # We need to install the headers before building the dependencies. If we
PATCH 14/26 xfs_healer: enable repairing filesystems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that our health monitoring daemon can initiate repairs. Because repairs can take a while to run, so we don't actually want to be doing that work in the event thread because the kernel queue can drop events if userspace doesn't respond in time.
Therefore, create a subprocess executor to run the repairs in the background, and do the repairs from there. The subprocess executor is similar in concept to what a libfrog workqueue does, but the workers do not share address space, which eliminates GIL contention.
--- healer/xfs_healer.py.in | 395 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 390 insertions(+), 5 deletions(-)
healer/xfs_healer.py.in +390 -5
@@ -19,6 +19,8 @@ import ctypes import pathlib import gc from concurrent.futures import ProcessPoolExecutor +import ctypes.util +from enum import Enum try: # Not all systems will have this json schema validation libarary,
@@ -71,6 +73,8 @@ log = False everything = False debug_fast = False printf_prefix = '' +want_repair = False +libhandle = None def printlogln(*args, **kwargs): '''Print a log message to stdout and flush it.'''
@@ -123,6 +127,9 @@ def _IOW(type, number, size): def _IOWR(type, number, size): return _IOC(_IOC_READ | _IOC_WRITE, type, number, size) +def _IOWR(type, number, size): + return _IOC(_IOC_READ | _IOC_WRITE, type, number, size) + # xfs health monitoring ioctl stuff XFS_HEALTH_MONITOR_FMT_JSON = 1 XFS_HEALTH_MONITOR_VERBOSE = 1 << 0
@@ -150,6 +157,238 @@ def open_health_monitor(fd, verbose = False): ret = fcntl.ioctl(fd, XFS_IOC_HEALTH_MONITOR, arg) return ret +# libhandle stuff +class xfs_fsid(ctypes.Structure): + _fields_ = [ + ("_val0", ctypes.c_uint), + ("_val1", ctypes.c_uint) + ] + +class xfs_fid(ctypes.Structure): + _fields_ = [ + ("fid_len", ctypes.c_ushort), + ("fid_pad", ctypes.c_ushort), + ("fid_gen", ctypes.c_uint), + ("fid_ino", ctypes.c_ulonglong) + ] + +class xfs_handle(ctypes.Structure): + _fields_ = [ + ("_ha_fsid", xfs_fsid), + ("ha_fid", xfs_fid) + ] +assert ctypes.sizeof(xfs_handle) == 24 + +class fshandle(object): + def __init__(self, fd, mountpoint): + global libhandle + global printf_prefix + + self.handle = xfs_handle() + + if mountpoint is None: + raise Exception(_('fshandle needs a mountpoint')) + + self.mountpoint = mountpoint + + # Create the file and fs handles for the open mountpoint + # so that we can compare them later + buf = ctypes.c_void_p() + buflen = ctypes.c_size_t() + ret = libhandle.fd_to_handle(fd, buf, buflen) + if ret < 0: + errcode = ctypes.get_errno() + errstr = os.strerror(errcode) + msg = _("cannot create handle") + raise OSError(errcode, f'{msg}: {errstr}', + printf_prefix) + expected_size = ctypes.sizeof(xfs_handle) + if buflen.value != expected_size: + libhandle.free_handle(buf, buflen.value) + msg = _("Bad file handle size") + raise Exception(f"{msg}: {buflen.value}") + + hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) + self.handle = hanp.contents + + def reopen(self): + '''Reopen a file handle obtained via weak reference.''' + global libhandle + global printf_prefix + + buf = ctypes.c_void_p() + buflen = ctypes.c_size_t() + + fd = os.open(self.mountpoint, os.O_RDONLY) + + # Create the file and fs handles for the open mountpoint + # so that we can compare them later + ret = libhandle.fd_to_handle(fd, buf, buflen) + if ret < 0: + errcode = ctypes.get_errno() + errstr = os.strerror(errcode) + os.close(fd) + msg = _("resampling handle") + raise OSError(errcode, f'{msg}: {errstr}', + printf_prefix) + + hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) + + # Did we get the same handle? + if buflen.value != ctypes.sizeof(xfs_handle) or \ + bytes(hanp.contents) != bytes(self.handle): + os.close(fd) + libhandle.free_handle(buf, buflen) + msg = _("reopening") + errstr = os.strerror(errno.ESTALE) + raise OSError(errno.ESTALE, f'{msg}: {errstr}', + printf_prefix) + + libhandle.free_handle(buf, buflen) + return fd + +def libhandle_load(): + '''Load libhandle and set things up.''' + global libhandle + + soname = ctypes.util.find_library('handle') + if soname is None: + errstr = os.strerror(errno.ENOENT) + msg = _("while finding library") + raise OSError(errno.ENOENT, f'{msg}: {errstr}', 'libhandle') + + libhandle = ctypes.CDLL(soname, use_errno = True) + libhandle.fd_to_handle.argtypes = ( + ctypes.c_int, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(ctypes.c_size_t)) + libhandle.handle_to_fshandle.argtypes = ( + ctypes.c_void_p, + ctypes.c_size_t, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(ctypes.c_size_t)) + libhandle.path_to_fshandle.argtypes = ( + ctypes.c_char_p, + ctypes.c_void_p, + ctypes.c_size_t) + libhandle.free_handle.argtypes = ( + ctypes.c_void_p, + ctypes.c_size_t) + +# metadata scrubbing stuff +XFS_SCRUB_TYPE_PROBE = 0 +XFS_SCRUB_TYPE_SB = 1 +XFS_SCRUB_TYPE_AGF = 2 +XFS_SCRUB_TYPE_AGFL = 3 +XFS_SCRUB_TYPE_AGI = 4 +XFS_SCRUB_TYPE_BNOBT = 5 +XFS_SCRUB_TYPE_CNTBT = 6 +XFS_SCRUB_TYPE_INOBT = 7 +XFS_SCRUB_TYPE_FINOBT = 8 +XFS_SCRUB_TYPE_RMAPBT = 9 +XFS_SCRUB_TYPE_REFCNTBT = 10 +XFS_SCRUB_TYPE_INODE = 11 +XFS_SCRUB_TYPE_BMBTD = 12 +XFS_SCRUB_TYPE_BMBTA = 13 +XFS_SCRUB_TYPE_BMBTC = 14 +XFS_SCRUB_TYPE_DIR = 15 +XFS_SCRUB_TYPE_XATTR = 16 +XFS_SCRUB_TYPE_SYMLINK = 17 +XFS_SCRUB_TYPE_PARENT = 18 +XFS_SCRUB_TYPE_RTBITMAP = 19 +XFS_SCRUB_TYPE_RTSUM = 20 +XFS_SCRUB_TYPE_UQUOTA = 21 +XFS_SCRUB_TYPE_GQUOTA = 22 +XFS_SCRUB_TYPE_PQUOTA = 23 +XFS_SCRUB_TYPE_FSCOUNTERS = 24 +XFS_SCRUB_TYPE_QUOTACHECK = 25 +XFS_SCRUB_TYPE_NLINKS = 26 +XFS_SCRUB_TYPE_HEALTHY = 27 +XFS_SCRUB_TYPE_DIRTREE = 28 +XFS_SCRUB_TYPE_METAPATH = 29 +XFS_SCRUB_TYPE_RGSUPER = 30 +XFS_SCRUB_TYPE_RGBITMAP = 31 +XFS_SCRUB_TYPE_RTRMAPBT = 32 +XFS_SCRUB_TYPE_RTREFCBT = 33 + +XFS_SCRUB_IFLAG_REPAIR = 1 << 0 +XFS_SCRUB_OFLAG_CORRUPT = 1 << 1 +XFS_SCRUB_OFLAG_PREEN = 1 << 2 +XFS_SCRUB_OFLAG_XFAIL = 1 << 3 +XFS_SCRUB_OFLAG_XCORRUPT = 1 << 4 +XFS_SCRUB_OFLAG_INCOMPLETE = 1 << 5 +XFS_SCRUB_OFLAG_WARNING = 1 << 6 +XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED = 1 << 7 +XFS_SCRUB_IFLAG_FORCE_REBUILD = 1 << 8 + +class xfs_scrub_metadata(ctypes.Structure): + _fields_ = [ + ('sm_type', ctypes.c_uint), + ('sm_flags', ctypes.c_uint), + ('sm_ino', ctypes.c_ulonglong), + ('sm_gen', ctypes.c_uint), + ('sm_agno', ctypes.c_uint), + ('_pad', ctypes.c_ulonglong * 5), + ] +assert ctypes.sizeof(xfs_scrub_metadata) == 64 + +XFS_IOC_SCRUB_METADATA = _IOWR(0x58, 60, xfs_scrub_metadata) + +def __xfs_repair_metadata(fd, type, group, ino, gen): + '''Call the kernel to repair some inode metadata.''' + + arg = xfs_scrub_metadata() + arg.sm_type = type + arg.sm_flags = XFS_SCRUB_IFLAG_REPAIR + arg.sm_ino = ino + arg.sm_gen = gen + arg.sm_agno = group + + fcntl.ioctl(fd, XFS_IOC_SCRUB_METADATA, arg) + return arg.sm_flags + +def xfs_repair_fs_metadata(fd, type): + '''Call the kernel to repair some whole-fs metadata.''' + return __xfs_repair_metadata(fd, type, 0, 0, 0) + +def xfs_repair_group_metadata(fd, type, group): + '''Call the kernel to repair some group metadata.''' + return __xfs_repair_metadata(fd, type, group, 0, 0) + +def xfs_repair_inode_metadata(fd, type, ino, gen): + '''Call the kernel to repair some inode metadata.''' + return __xfs_repair_metadata(fd, type, 0, ino, gen) + +class RepairOutcome(Enum): + Success = 1, + Unnecessary = 2, + MightBeOk = 3, + Failed = 4, + + def from_oflags(oflags): + '''Translate scrub output flags to outcome.''' + if oflags & (XFS_SCRUB_OFLAG_CORRUPT | \ + XFS_SCRUB_OFLAG_INCOMPLETE): + return RepairOutcome.Failed + + if oflags & XFS_SCRUB_OFLAG_XFAIL: + return RepairOutcome.MightBeOk + + if oflags & XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED: + return RepairOutcome.Unnecessary + + return RepairOutcome.Success + + def report(self): + if self == RepairOutcome.Failed: + return _("Repair unsuccessful; offline repair required.") + if self == RepairOutcome.MightBeOk: + return _("Seems correct but cross-referencing failed; offline repair recommended.") + if self == RepairOutcome.Unnecessary: + return _("No modification needed.") + if self == RepairOutcome.Success: + return _("Repairs successful.") + # main program def health_reports(mon_fp):
@@ -263,7 +502,7 @@ def report_shutdown(event): msg = _("filesystem shut down due to") printlogln(f"{printf_prefix}: {msg} {some_reasons}") -def handle_event(lines): +def handle_event(lines, fh): '''Handle an event asynchronously.''' global log
@@ -306,17 +545,23 @@ def handle_event(lines): except Exception as e: eprintln(f"event reporting: {e}") + if want_repair and event['type'] == 'sick': + repair_metadata(event, fh) + def monitor(mountpoint, event_queue, **kwargs): '''Monitor the given mountpoint for health events.''' global everything + global log + global printf_prefix + global want_repair - def event_loop(mon_fd, event_queue): + def event_loop(mon_fd, event_queue, fh): # Ownership of mon_fd (and hence responsibility for closing it) # is transferred to the mon_fp object. with os.fdopen(mon_fd) as mon_fp: nr = 0 for lines in health_reports(mon_fp): - event_queue.submit(handle_event, lines) + event_queue.submit(handle_event, lines, fh) # Periodically run the garbage collector to # constrain memory usage in the main thread.
@@ -332,6 +577,13 @@ def monitor(mountpoint, event_queue, **kwargs): eprintln(f"{mountpoint}: {e}") return 1 + try: + fh = fshandle(fd, mountpoint) if want_repair else None + except Exception as e: + eprintln(f"{mountpoint}: {e}") + os.close(fd) + return 1 + try: mon_fd = open_health_monitor(fd, verbose = everything) except OSError as e:
@@ -345,18 +597,140 @@ def monitor(mountpoint, event_queue, **kwargs): eprintln(f"{mountpoint}: {e}") return 1 finally: - # Close the mountpoint if opening the health monitor fails + # Close the mountpoint if opening the health monitor fails; + # the handle object will free its own memory. os.close(fd) try: # mon_fd is consumed by this function - event_loop(mon_fd, event_queue) + event_loop(mon_fd, event_queue, fh) except Exception as e: eprintln(f"{mountpoint}: {e}") return 1 return 0 +def __scrub_type(code): + '''Convert a "structures" json list to a scrub type code.''' + SCRUB_TYPES = { + "probe": XFS_SCRUB_TYPE_PROBE, + "sb": XFS_SCRUB_TYPE_SB, + "agf": XFS_SCRUB_TYPE_AGF, + "agfl": XFS_SCRUB_TYPE_AGFL, + "agi": XFS_SCRUB_TYPE_AGI, + "bnobt": XFS_SCRUB_TYPE_BNOBT, + "cntbt": XFS_SCRUB_TYPE_CNTBT, + "inobt": XFS_SCRUB_TYPE_INOBT, + "finobt": XFS_SCRUB_TYPE_FINOBT, + "rmapbt": XFS_SCRUB_TYPE_RMAPBT, + "refcountbt": XFS_SCRUB_TYPE_REFCNTBT, + "inode": XFS_SCRUB_TYPE_INODE, + "bmapbtd": XFS_SCRUB_TYPE_BMBTD, + "bmapbta": XFS_SCRUB_TYPE_BMBTA, + "bmapbtc": XFS_SCRUB_TYPE_BMBTC, + "directory": XFS_SCRUB_TYPE_DIR, + "xattr": XFS_SCRUB_TYPE_XATTR, + "symlink": XFS_SCRUB_TYPE_SYMLINK, + "parent": XFS_SCRUB_TYPE_PARENT, + "rtbitmap": XFS_SCRUB_TYPE_RTBITMAP, + "rtsummary": XFS_SCRUB_TYPE_RTSUM, + "usrquota": XFS_SCRUB_TYPE_UQUOTA, + "grpquota": XFS_SCRUB_TYPE_GQUOTA, + "prjquota": XFS_SCRUB_TYPE_PQUOTA, + "fscounters": XFS_SCRUB_TYPE_FSCOUNTERS, + "quotacheck": XFS_SCRUB_TYPE_QUOTACHECK, + "nlinks": XFS_SCRUB_TYPE_NLINKS, + "healthy": XFS_SCRUB_TYPE_HEALTHY, + "dirtree": XFS_SCRUB_TYPE_DIRTREE, + "metapath": XFS_SCRUB_TYPE_METAPATH, + "rgsuper": XFS_SCRUB_TYPE_RGSUPER, + "rgbitmap": XFS_SCRUB_TYPE_RGBITMAP, + "rtrmapbt": XFS_SCRUB_TYPE_RTRMAPBT, + "rtrefcountbt": XFS_SCRUB_TYPE_RTREFCBT, + } + + if code not in SCRUB_TYPES: + return None + + return SCRUB_TYPES[code] + +def repair_wholefs(event, fd): + '''React to a fs-domain corruption event by repairing it.''' + for structure in event['structures']: + struct = _(structure) + scrub_type = __scrub_type(structure) + if scrub_type is None: + continue + try: + oflags = xfs_repair_fs_metadata(fd, scrub_type) + outcome = RepairOutcome.from_oflags(oflags) + report = outcome.report() + printlogln(f"{printf_prefix}: {struct}: {report}") + except Exception as e: + eprintln(f"{printf_prefix}: {struct}: {e}") + +def repair_group(event, fd, group_type): + '''React to a group-domain corruption event by repairing it.''' + for structure in event['structures']: + struct = _(structure) + scrub_type = __scrub_type(structure) + if scrub_type is None: + continue + try: + oflags = xfs_repair_group_metadata(fd, scrub_type, + event['group']) + outcome = RepairOutcome.from_oflags(oflags) + report = outcome.report() + printlogln(f"{printf_prefix}: {struct}: {report}") + except Exception as e: + eprintln(f"{printf_prefix}: {struct}: {e}") + +def repair_inode(event, fd): + '''React to a inode-domain corruption event by repairing it.''' + for structure in event['structures']: + struct = _(structure) + scrub_type = __scrub_type(structure) + if scrub_type is None: + continue + try: + oflags = xfs_repair_inode_metadata(fd, scrub_type, + event['inumber'], event['generation']) + outcome = RepairOutcome.from_oflags(oflags) + report = outcome.report() + printlogln(f"{printf_prefix}: {struct}: {report}") + except Exception as e: + eprintln(f"{printf_prefix}: {struct}: {e}") + +def repair_metadata(event, fh): + '''Repair a metadata corruption.''' + global debug + global printf_prefix + + if debug: + printlogln(f'repair {event}') + + try: + fd = fh.reopen() + except Exception as e: + eprintln(f"{printf_prefix}: {e}") + return + + try: + if event['domain'] in ['fs', 'realtime']: + repair_wholefs(event, fd) + elif event['domain'] in ['perag', 'rtgroup']: + repair_group(event, fd, event['domain']) + elif event['domain'] == 'inode': + repair_inode(event, fd) + else: + domain = event['domain'] + msg = _("Unknown metadata domain") + raise Exception(f"{msg} \"{domain}\".") + except Exception as e: + eprintln(f"{printf_prefix}: {e}") + finally: + os.close(fd) + def main(): global debug global log
@@ -364,6 +738,7 @@ def main(): global everything global debug_fast global validator_fn + global want_repair parser = argparse.ArgumentParser( \ description = _("Automatically heal damage to XFS filesystem metadata"))
@@ -384,6 +759,8 @@ def main(): parser.add_argument('--event-schema', type = str, \ default = '@pkg_data_dir@/xfs_healthmon.schema.json', \ help = argparse.SUPPRESS) + parser.add_argument("--repair", action = "store_true", \ + help = _("Always repair corrupt metadata")) args = parser.parse_args() if args.V:
@@ -400,6 +777,12 @@ def main(): if not validator_fn: return 1 + try: + libhandle_load() + except OSError as e: + eprintln(f"libhandle: {e}") + return 1 + if args.debug: debug = True if args.log:
@@ -408,6 +791,8 @@ def main(): everything = True if args.debug_fast: debug_fast = True + if args.repair: + want_repair = True # Use a separate subprocess to handle the events so that the main event # reading process does not block on the GIL of the event handling
PATCH 15/26 xfs_healer: check for fs features needed for effective repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Online repair relies heavily on back references such as reverse mappings and directory parent pointers to add redundancy to the filesystem. Check for these two features and whine a bit if they are missing.
--- healer/xfs_healer.py.in | 92 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+)
healer/xfs_healer.py.in +92 -0
@@ -74,6 +74,8 @@ everything = False debug_fast = False printf_prefix = '' want_repair = False +has_parent = False +has_rmapbt = False libhandle = None def printlogln(*args, **kwargs):
@@ -347,6 +349,19 @@ def __xfs_repair_metadata(fd, type, group, ino, gen): fcntl.ioctl(fd, XFS_IOC_SCRUB_METADATA, arg) return arg.sm_flags +def xfs_repair_is_supported(fd): + '''Ask the kernel if it supports repairs.''' + + arg = xfs_scrub_metadata() + arg.sm_type = XFS_SCRUB_TYPE_PROBE + arg.sm_flags = XFS_SCRUB_IFLAG_REPAIR + + try: + fcntl.ioctl(fd, XFS_IOC_SCRUB_METADATA, arg) + except: + return False + return True + def xfs_repair_fs_metadata(fd, type): '''Call the kernel to repair some whole-fs metadata.''' return __xfs_repair_metadata(fd, type, 0, 0, 0)
@@ -389,6 +404,57 @@ class RepairOutcome(Enum): if self == RepairOutcome.Success: return _("Repairs successful.") +# fsgeometry ioctl +class xfs_fsop_geom(ctypes.Structure): + _fields_ = [ + ("blocksize", ctypes.c_uint), + ("rtextesize", ctypes.c_uint), + ("agblocks", ctypes.c_uint), + ("agcount", ctypes.c_uint), + ("logblocks", ctypes.c_uint), + ("sectsize", ctypes.c_uint), + ("inodesize", ctypes.c_uint), + ("imaxpct", ctypes.c_uint), + ("datablocks", ctypes.c_ulonglong), + ("rtblocks", ctypes.c_ulonglong), + ("rtextents", ctypes.c_ulonglong), + ("logstart", ctypes.c_ulonglong), + ("uuid", ctypes.c_ubyte * 16), + ("sunit", ctypes.c_uint), + ("swidth", ctypes.c_uint), + ("version", ctypes.c_uint), + ("flags", ctypes.c_uint), + ("logsectsize", ctypes.c_uint), + ("rtsectsize", ctypes.c_uint), + ("dirblocksize", ctypes.c_uint), + ("logsunit", ctypes.c_uint), + ("sick", ctypes.c_uint), + ("checked", ctypes.c_uint), + ("rgblocks", ctypes.c_uint), + ("rgcount", ctypes.c_uint), + ("_pad", ctypes.c_ulonglong * 16), + ] +assert ctypes.sizeof(xfs_fsop_geom) == 256 + +XFS_FSOP_GEOM_FLAGS_RMAPBT = 1 << 19 +XFS_FSOP_GEOM_FLAGS_PARENT = 1 << 25 + +XFS_IOC_FSGEOMETRY = _IOR (0x58, 126, xfs_fsop_geom) + +def xfs_has_parent(fd): + '''Does this filesystem have parent pointers?''' + + arg = xfs_fsop_geom() + fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) + return arg.flags & XFS_FSOP_GEOM_FLAGS_PARENT != 0 + +def xfs_has_rmapbt(fd): + '''Does this filesystem have reverse mapping?''' + + arg = xfs_fsop_geom() + fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) + return arg.flags & XFS_FSOP_GEOM_FLAGS_RMAPBT != 0 + # main program def health_reports(mon_fp):
@@ -554,6 +620,8 @@ def monitor(mountpoint, event_queue, **kwargs): global log global printf_prefix global want_repair + global has_parent + global has_rmapbt def event_loop(mon_fd, event_queue, fh): # Ownership of mon_fd (and hence responsibility for closing it)
@@ -577,6 +645,30 @@ def monitor(mountpoint, event_queue, **kwargs): eprintln(f"{mountpoint}: {e}") return 1 + try: + has_parent = xfs_has_parent(fd) + has_rmapbt = xfs_has_rmapbt(fd) + except Exception as e: + # Don't care if we can't detect parent pointers or rmap + msg = _("detecting fs features") + eprintln(_(f'{printf_prefix}: {msg}: {e}')) + + # Check that the kernel supports repairs at all. + if want_repair and not xfs_repair_is_supported(fd): + msg = _("XFS online repair is not supported, exiting") + printlogln(f"{mountpoint}: {msg}") + os.close(fd) + return 1 + + # Check for the backref metadata that makes repair effective. + if want_repair: + if not has_rmapbt: + msg = _("XFS online repair is less effective without rmap btrees.") + printlogln(f"{mountpoint}: {msg}") + if not has_parent: + msg = _("XFS online repair is less effective without parent pointers.") + printlogln(f"{mountpoint}: {msg}") + try: fh = fshandle(fd, mountpoint) if want_repair else None except Exception as e:
PATCH 16/26 xfs_healer: use getparents to look up file names
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the kernel tells about something that happened to a file, use the GETPARENTS ioctl to try to look up the path to that file for more ergonomic reporting.
--- healer/xfs_healer.py.in | 248 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 242 insertions(+), 6 deletions(-)
healer/xfs_healer.py.in +242 -6
@@ -21,6 +21,7 @@ import gc from concurrent.futures import ProcessPoolExecutor import ctypes.util from enum import Enum +import collections try: # Not all systems will have this json schema validation libarary,
@@ -182,12 +183,18 @@ class xfs_handle(ctypes.Structure): assert ctypes.sizeof(xfs_handle) == 24 class fshandle(object): - def __init__(self, fd, mountpoint): + def __init__(self, fd, mountpoint = None): global libhandle global printf_prefix self.handle = xfs_handle() + if isinstance(fd, fshandle): + # copy an existing fshandle + self.mountpoint = fd.mountpoint + ctypes.pointer(self.handle)[0] = fd.handle + return + if mountpoint is None: raise Exception(_('fshandle needs a mountpoint'))
@@ -249,6 +256,11 @@ class fshandle(object): libhandle.free_handle(buf, buflen) return fd + def subst(self, ino, gen): + '''Substitute the inode and generation components of a handle.''' + self.handle.ha_fid.fid_ino = ino + self.handle.ha_fid.fid_gen = gen + def libhandle_load(): '''Load libhandle and set things up.''' global libhandle
@@ -455,6 +467,170 @@ def xfs_has_rmapbt(fd): fcntl.ioctl(fd, XFS_IOC_FSGEOMETRY, arg) return arg.flags & XFS_FSOP_GEOM_FLAGS_RMAPBT != 0 +# getparents ioctl +class xfs_attrlist_cursor(ctypes.Structure): + _fields_ = [ + ("_opaque0", ctypes.c_uint), + ("_opaque1", ctypes.c_uint), + ("_opaque2", ctypes.c_uint), + ("_opaque3", ctypes.c_uint) + ] + +class xfs_getparents_rec(ctypes.Structure): + _fields_ = [ + ("gpr_parent", xfs_handle), + ("gpr_reclen", ctypes.c_uint), + ("_gpr_reserved", ctypes.c_uint), + ] + +xfs_getparents_tuple = collections.namedtuple('xfs_getparents_tuple', \ + ['gpr_parent', 'gpr_reclen', 'gpr_name']) + +class xfs_getparents_rec_array(object): + def __init__(self, nr_bytes): + self.nr_bytes = nr_bytes + self.bytearray = (ctypes.c_byte * int(nr_bytes))() + + def __slice_to_record(self, bufslice): + '''Compute the number of bytes in a getparents record that contain a null-terminated directory entry name.''' + rec = ctypes.cast(bytes(bufslice), \ + ctypes.POINTER(xfs_getparents_rec)) + fixedlen = ctypes.sizeof(xfs_getparents_rec) + namelen = rec.contents.gpr_reclen - fixedlen + + for i in range(0, namelen): + if bufslice[fixedlen + i] == 0: + namelen = i + break + + if namelen == 0: + return + + return xfs_getparents_tuple( + gpr_parent = rec.contents.gpr_parent, + gpr_reclen = rec.contents.gpr_reclen, + gpr_name = bufslice[fixedlen:fixedlen + namelen]) + + def get_buffer(self): + '''Return a pointer to the bytearray masquerading as an int.''' + return ctypes.addressof(self.bytearray) + + def __iter__(self): + '''Walk the getparents records in this array.''' + off = 0 + nr = 0 + buf = bytes(self.bytearray) + while off < self.nr_bytes: + bufslice = buf[off:] + t = self.__slice_to_record(bufslice) + if t is None: + break + yield t + off += t.gpr_reclen + nr += 1 + +class xfs_getparents(ctypes.Structure): + _fields_ = [ + ("_gp_cursor", xfs_attrlist_cursor), + ("gp_iflags", ctypes.c_ushort), + ("gp_oflags", ctypes.c_ushort), + ("gp_bufsize", ctypes.c_uint), + ("_pad", ctypes.c_ulonglong), + ("gp_buffer", ctypes.c_ulonglong) + ] + + def __init__(self, fd, nr_bytes): + self.fd = fd + self.records = xfs_getparents_rec_array(nr_bytes) + self.gp_buffer = self.records.get_buffer() + self.gp_bufsize = nr_bytes + + def __call_kernel(self): + if self.gp_oflags & XFS_GETPARENTS_OFLAG_DONE: + return False + + ret = fcntl.ioctl(self.fd, XFS_IOC_GETPARENTS, self) + if ret != 0: + return False + + return self.gp_oflags & XFS_GETPARENTS_OFLAG_ROOT == 0 + + def __iter__(self): + ctypes.memset(ctypes.pointer(self._gp_cursor), 0, \ + ctypes.sizeof(xfs_attrlist_cursor)) + + while self.__call_kernel(): + for i in self.records: + yield i + +class xfs_getparents_by_handle(ctypes.Structure): + _fields_ = [ + ("gph_handle", xfs_handle), + ("gph_request", xfs_getparents) + ] + + def __init__(self, fd, fh, nr_bytes): + self.fd = fd + self.records = xfs_getparents_rec_array(nr_bytes) + self.gph_request.gp_buffer = self.records.get_buffer() + self.gph_request.gp_bufsize = nr_bytes + self.gph_handle = fh.handle + + def __call_kernel(self): + if self.gph_request.gp_oflags & XFS_GETPARENTS_OFLAG_DONE: + return False + + ret = fcntl.ioctl(self.fd, XFS_IOC_GETPARENTS_BY_HANDLE, self) + if ret != 0: + return False + + return self.gph_request.gp_oflags & XFS_GETPARENTS_OFLAG_ROOT == 0 + + def __iter__(self): + ctypes.memset(ctypes.pointer(self.gph_request._gp_cursor), 0, \ + ctypes.sizeof(xfs_attrlist_cursor)) + while self.__call_kernel(): + for i in self.records: + yield i + +assert ctypes.sizeof(xfs_getparents) == 40 +assert ctypes.sizeof(xfs_getparents_by_handle) == 64 +assert ctypes.sizeof(xfs_getparents_rec) == 32 + +XFS_GETPARENTS_OFLAG_ROOT = 1 << 0 +XFS_GETPARENTS_OFLAG_DONE = 1 << 1 + +XFS_IOC_GETPARENTS = _IOWR(0x58, 62, xfs_getparents) +XFS_IOC_GETPARENTS_BY_HANDLE = _IOWR(0x58, 63, xfs_getparents_by_handle) + +def fgetparents(fd, fh = None, bufsize = 1024): + '''Return all the parent pointers for a given fd and/or handle.''' + + if fh is not None: + return xfs_getparents_by_handle(fd, fh, bufsize) + return xfs_getparents(fd, bufsize) + +def fgetpath(fd, fh = None, mountpoint = None): + '''Return a list of path components up to the root dir of the filesystem for a given fd.''' + ret = [] + if fh is None: + nfh = fshandle(fd, mountpoint) + else: + # Don't subst into the caller's handle + nfh = fshandle(fh) + + while True: + added = False + for pptr in fgetparents(fd, nfh): + ret.insert(0, pptr.gpr_name) + nfh.subst(pptr.gpr_parent.ha_fid.fid_ino, \ + pptr.gpr_parent.ha_fid.fid_gen) + added = True + break + if not added: + break + return ret + # main program def health_reports(mon_fp):
@@ -479,14 +655,29 @@ def health_reports(mon_fp): lines = [] buf = mon_fp.readline() +def file_event_to_prefix(event): + '''Compute the logging prefix for this event.''' + global printf_prefix + + if 'path' in event: + path = event['path'] + return f"{printf_prefix}{os.sep}{path}:" + + inumber = event['inumber'] + igen = event['generation'] + inom = _("ino") + igenm = _("gen") + return f"{printf_prefix}: {inom} {inumber} {igenm} {igen:#x}" + def report_event(event): '''Log a monitoring event to stdout.''' global printf_prefix if event['domain'] == 'inode': + prefix = file_event_to_prefix(event) structures = ', '.join([_(x) for x in event['structures']]) status = _(event['type']) - printlogln(f"{printf_prefix}: {structures} {status}") + printlogln(f"{prefix} {structures} {status}") elif event['domain'] == 'perag': structures = ', '.join([_(x) for x in event['structures']])
@@ -517,12 +708,13 @@ def report_event(event): printlogln(f"{printf_prefix}: {msg} {device} {daddrm} {daddr:#x} {bbcountm} {bbcount:#x}") elif event['domain'] == 'filerange': + prefix = file_event_to_prefix(event) event_type = _(event['type']) pos = event['pos'] length = event['length'] posm = _("pos") lenm = _("len") - printlogln(f"{printf_prefix}: {event_type} {posm} {pos} {lenm} {length}") + printlogln(f"{prefix} {event_type} pos {pos} len {length}") def report_lost(event): '''Report that the kernel lost events.'''
@@ -571,6 +763,40 @@ def report_shutdown(event): def handle_event(lines, fh): '''Handle an event asynchronously.''' global log + global has_parent + + def pathify_event(event, fh): + '''Come up with a directory tree path for a file event.''' + try: + path_fd = fh.reopen() + except Exception as e: + # Not the end of the world if we get nothing + return + + try: + fh2 = fshandle(fh) + except OSError as e: + if e.errno != errno.EOPNOTSUPP: + msg = _("making new file handle") + eprintln(f'{printf_prefix}: {msg}: {e}') + os.close(path_fd) + return + except Exception as e: + # Not the end of the world if we get nothing + os.close(path_fd) + return + + try: + fh2.subst(event['inumber'], event['generation']) + components = [x.decode('utf-8') for x in fgetpath(path_fd, fh2)] + event['path'] = os.sep.join(components) + except Exception as e: + # Path walking might be unavailable if the directory + # tree is corrupt. Since this is optional, we don't + # report anything. + pass + finally: + os.close(path_fd) # Convert array of strings into a json object try:
@@ -605,13 +831,21 @@ def handle_event(lines, fh): return # Deal with everything else. + maybe_pathify = event['domain'] in ('inode', 'filerange') and has_parent if log: + if maybe_pathify and not debug_fast: + pathify_event(event, fh) + maybe_pathify = False + try: report_event(event) except Exception as e: eprintln(f"event reporting: {e}") if want_repair and event['type'] == 'sick': + if maybe_pathify: + pathify_event(event, fh) + repair_metadata(event, fh) def monitor(mountpoint, event_queue, **kwargs):
@@ -670,7 +904,7 @@ def monitor(mountpoint, event_queue, **kwargs): printlogln(f"{mountpoint}: {msg}") try: - fh = fshandle(fd, mountpoint) if want_repair else None + fh = fshandle(fd, mountpoint) if want_repair or has_parent else None except Exception as e: eprintln(f"{mountpoint}: {e}") os.close(fd)
@@ -779,6 +1013,8 @@ def repair_group(event, fd, group_type): def repair_inode(event, fd): '''React to a inode-domain corruption event by repairing it.''' + prefix = file_event_to_prefix(event) + for structure in event['structures']: struct = _(structure) scrub_type = __scrub_type(structure)
@@ -789,9 +1025,9 @@ def repair_inode(event, fd): event['inumber'], event['generation']) outcome = RepairOutcome.from_oflags(oflags) report = outcome.report() - printlogln(f"{printf_prefix}: {struct}: {report}") + printlogln(f"{prefix} {structure}: {report}") except Exception as e: - eprintln(f"{printf_prefix}: {struct}: {e}") + eprintln(f"{prefix} {structure}: {e}") def repair_metadata(event, fh): '''Repair a metadata corruption.'''
PATCH 17/26 builddefs: refactor udev directory specification
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Refactor the code that finds the udev rules directory to detect the location of the parent udev directory instead. IOWs, we go from:
UDEV_RULE_DIR=/foo/bar/rules.d
to:
UDEV_DIR=/foo/bar
UDEV_RULE_DIR=/foo/bar/rules.d
This is needed by the next patch, which adds a helper script.
--- configure.ac | 2 +- include/builddefs.in | 3 ++- m4/package_services.m4 | 30 +++++++++++++++--------------- 3 files changed, 18 insertions(+), 17 deletions(-)
configure.ac +1 -1
@@ -184,7 +184,7 @@ if test "$enable_scrub" = "yes"; then fi AC_CONFIG_SYSTEMD_SYSTEM_UNIT_DIR AC_CONFIG_CROND_DIR -AC_CONFIG_UDEV_RULE_DIR +AC_CONFIG_UDEV_DIR AC_HAVE_BLKID_TOPO AC_HAVE_TRIVIAL_AUTO_VAR_INIT AC_STRERROR_R_RETURNS_STRING
include/builddefs.in +2 -1
@@ -116,7 +116,8 @@ SYSTEMD_SYSTEM_UNIT_DIR = @systemd_system_unit_dir@ HAVE_CROND = @have_crond@ CROND_DIR = @crond_dir@ HAVE_UDEV = @have_udev@ -UDEV_RULE_DIR = @udev_rule_dir@ +UDEV_DIR = @udev_dir@ +UDEV_RULE_DIR = @udev_dir@/rules.d HAVE_LIBURCU_ATOMIC64 = @have_liburcu_atomic64@ STRERROR_R_RETURNS_STRING = @strerror_r_returns_string@
m4/package_services.m4 +15 -15
@@ -77,33 +77,33 @@ AC_DEFUN([AC_CONFIG_CROND_DIR], ]) # -# Figure out where to put udev rule files +# Figure out where to put udev files # -AC_DEFUN([AC_CONFIG_UDEV_RULE_DIR], +AC_DEFUN([AC_CONFIG_UDEV_DIR], [ AC_REQUIRE([PKG_PROG_PKG_CONFIG]) - AC_ARG_WITH([udev_rule_dir], - [AS_HELP_STRING([--with-udev-rule-dir@<:@=DIR@:>@], - [Install udev rules into DIR.])], + AC_ARG_WITH([udev_dir], + [AS_HELP_STRING([--with-udev-dir@<:@=DIR@:>@], + [Install udev files underneath DIR.])], [], - [with_udev_rule_dir=yes]) - AS_IF([test "x${with_udev_rule_dir}" != "xno"], + [with_udev_dir=yes]) + AS_IF([test "x${with_udev_dir}" != "xno"], [ - AS_IF([test "x${with_udev_rule_dir}" = "xyes"], + AS_IF([test "x${with_udev_dir}" = "xyes"], [ PKG_CHECK_MODULES([udev], [udev], [ - with_udev_rule_dir="$($PKG_CONFIG --variable=udev_dir udev)/rules.d" + with_udev_dir="$($PKG_CONFIG --variable=udev_dir udev)" ], [ - with_udev_rule_dir="" + with_udev_dir="" ]) m4_pattern_allow([^PKG_(MAJOR|MINOR|BUILD|REVISION)$]) ]) - AC_MSG_CHECKING([for udev rule dir]) - udev_rule_dir="${with_udev_rule_dir}" - AS_IF([test -n "${udev_rule_dir}"], + AC_MSG_CHECKING([for udev dir]) + udev_dir="${with_udev_dir}" + AS_IF([test -n "${udev_dir}"], [ - AC_MSG_RESULT(${udev_rule_dir}) + AC_MSG_RESULT(${udev_dir}) have_udev="yes" ], [
@@ -115,5 +115,5 @@ AC_DEFUN([AC_CONFIG_UDEV_RULE_DIR], have_udev="disabled" ]) AC_SUBST(have_udev) - AC_SUBST(udev_rule_dir) + AC_SUBST(udev_dir) ])
PATCH 18/26 xfs_healer: create a background monitoring service
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a systemd service for our self-healing service and activate it automatically.
--- healer/Makefile | 34 ++++++++++++- healer/system-xfs_healer.slice | 31 ++++++++++++ healer/xfs_healer.py.in | 9 +++ healer/xfs_healer.rules | 7 +++ healer/[email protected] | 107 ++++++++++++++++++++++++++++++++++++++++ healer/xfs_healer_start | 17 ++++++ 6 files changed, 204 insertions(+), 1 deletion(-) create mode 100644 healer/system-xfs_healer.slice create mode 100644 healer/xfs_healer.rules create mode 100644 healer/[email protected] create mode 100755 healer/xfs_healer_start
healer/Makefile +33 -1
@@ -9,9 +9,24 @@ include $(builddefs) XFS_HEALER_PROG = xfs_healer.py INSTALL_HEALER = install-healer +ifeq ($(HAVE_SYSTEMD),yes) +INSTALL_HEALER += install-systemd +SYSTEMD_SERVICES=\ + system-xfs_healer.slice \ + [email protected] +OPTIONAL_TARGETS += $(SYSTEMD_SERVICES) + +ifeq ($(HAVE_UDEV),yes) + UDEV_RULES = xfs_healer.rules + XFS_HEALER_HELPER = xfs_healer_start + INSTALL_HEALER += install-udev + OPTIONAL_TARGETS += $(XFS_HEALER_HELPER) +endif +endif + LDIRT = $(XFS_HEALER_PROG) -default: $(XFS_HEALER_PROG) +default: $(XFS_HEALER_PROG) $(SYSTEMD_SERVICES) $(UDEV_RULES) $(XFS_HEALER_HELPER) $(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext.py @echo " [SED] $@"
@@ -22,6 +37,11 @@ $(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext < $< > $@ $(Q)chmod a+x $@ +%.service: %.service.in $(builddefs) + @echo " [SED] $@" + $(Q)$(SED) -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ + < $< > $@ + include $(BUILDRULES) install: $(INSTALL_HEALER)
@@ -30,6 +50,18 @@ install-healer: default $(INSTALL) -m 755 -d $(PKG_LIBEXEC_DIR) $(INSTALL) -m 755 $(XFS_HEALER_PROG) $(PKG_LIBEXEC_DIR)/xfs_healer +install-systemd: default + $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) + $(INSTALL) -m 644 $(SYSTEMD_SERVICES) $(SYSTEMD_SYSTEM_UNIT_DIR) + +install-udev: default + $(INSTALL) -m 755 -d $(UDEV_DIR) + $(INSTALL) -m 755 $(XFS_HEALER_HELPER) $(UDEV_DIR) + $(INSTALL) -m 755 -d $(UDEV_RULE_DIR) + for i in $(UDEV_RULES); do \ + $(INSTALL) -m 644 $$i $(UDEV_RULE_DIR)/64-$$i; \ + done + install-dev: -include .dep
healer/system-xfs_healer.slice new file +31 -0
@@ -0,0 +1,31 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# Author: Darrick J. Wong <[email protected]> + +[Unit] +Description=xfs_healer background service slice +Before=slices.target + +[Slice] + +# If the CPU usage cgroup controller is available, don't use more than 2 cores +# for all background processes. One thread to read events, another to run +# repairs. +CPUQuota=200% +CPUAccounting=true + +[Install] +# As of systemd 249, the systemd cgroupv2 configuration code will drop resource +# controllers from the root and system.slice cgroups at startup if it doesn't +# find any direct dependencies that require a given controller. Newly +# activated units with resource control directives are created under the system +# slice but do not cause a reconfiguration of the slice's resource controllers. +# Hence we cannot put CPUQuota= into the xfs_healer service units directly. +# +# For the CPUQuota directive to have any effect, we must therefore create an +# explicit definition file for the slice that systemd creates to contain the +# xfs_healer instance units (e.g. [email protected]) and we must configure +# this slice as a dependency of the system slice to establish the direct +# dependency relation. +WantedBy=system.slice
healer/xfs_healer.py.in +9 -0
@@ -22,6 +22,7 @@ from concurrent.futures import ProcessPoolExecutor import ctypes.util from enum import Enum import collections +import time try: # Not all systems will have this json schema validation libarary,
@@ -1137,6 +1138,14 @@ def main(): pass args.event_queue.shutdown() + + # See the service mode comments in xfs_scrub.c for why we sleep and + # compress all nonzero exit codes to 1. + if 'SERVICE_MODE' in os.environ: + time.sleep(2) + if ret != 0: + ret = 1 + return ret if __name__ == '__main__':
healer/xfs_healer.rules new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (c) 2024-2025 Oracle. All rights reserved. +# Author: Darrick J. Wong <[email protected]> +# +# Start autonomous self healing automatically +ACTION=="add", SUBSYSTEM=="xfs", ENV{TYPE}=="mount", RUN+="xfs_healer_start"
healer/[email protected] new file +107 -0
107 lines not on this page · raw message
healer/xfs_healer_start new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh + +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# Author: Darrick J. Wong <[email protected]> + +# Start the xfs_healer service when the filesystem is mounted + +command -v systemctl || exit 0 + +grep "^$SOURCE[[:space:]]" /proc/mounts | while read source mntpt therest; do + inst="$(systemd-escape --path "$mntpt")" + systemctl restart --no-block "xfs_healer@$inst" && break +done + +exit 0
PATCH 19/26 xfs_healer: don't start service if kernel support unavailable
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use ExecCondition= in the system service to check if kernel support for the health monitor is available. If not, we don't want to run the service, have it fail, and generate a bunch of silly log messages.
--- healer/xfs_healer.py.in | 30 +++++++++++++++++++++++++++++- healer/[email protected] | 1 + 2 files changed, 30 insertions(+), 1 deletion(-)
healer/xfs_healer.py.in +29 -1
@@ -849,7 +849,27 @@ def handle_event(lines, fh): repair_metadata(event, fh) -def monitor(mountpoint, event_queue, **kwargs): +def check_monitor(mountpoint, fd): + '''Check if the kernel can send us health events for the given mountpoint.''' + + try: + mon_fd = open_health_monitor(fd, verbose = everything) + except OSError as e: + # Error opening monitor (or it's simply not there); monitor + # not available. + if e.errno == errno.ENOTTY or e.errno == errno.EOPNOTSUPP: + msg = _("XFS health monitoring not supported.") + eprintln(f"{mountpoint}: {msg}") + os.close(fd) + return 1 + + os.close(mon_fd) + os.close(fd) + + # Monitor available; success! + return 0 + +def monitor(mountpoint, event_queue, check, **kwargs): '''Monitor the given mountpoint for health events.''' global everything global log
@@ -904,6 +924,12 @@ def monitor(mountpoint, event_queue, **kwargs): msg = _("XFS online repair is less effective without parent pointers.") printlogln(f"{mountpoint}: {msg}") + # Now that we know that we can repair if the user wanted to, make sure + # that the kernel supports reporting events if that was as far as the + # user wanted us to go. + if check: + return check_monitor(mountpoint, fd) + try: fh = fshandle(fd, mountpoint) if want_repair or has_parent else None except Exception as e:
@@ -1090,6 +1116,8 @@ def main(): help = argparse.SUPPRESS) parser.add_argument("--repair", action = "store_true", \ help = _("Always repair corrupt metadata")) + parser.add_argument("--check", action = "store_true", \ + help = _("Check that health monitoring is supported")) args = parser.parse_args() if args.V:
healer/[email protected] +1 -0
@@ -17,6 +17,7 @@ RequiresMountsFor=%f [Service] Type=exec Environment=SERVICE_MODE=1 +ExecCondition=@pkg_libexec_dir@/xfs_healer --check %f ExecStart=@pkg_libexec_dir@/xfs_healer --log %f SyslogIdentifier=%N
PATCH 20/26 xfs_healer: use the autofsck fsproperty to select mode
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make the xfs_healer background service query the autofsck filesystem property to figure out which operating mode it should use.
--- healer/xfs_healer.py.in | 75 +++++++++++++++++++++++++++++++++++++++-- healer/[email protected] | 2 + 2 files changed, 72 insertions(+), 5 deletions(-)
healer/xfs_healer.py.in +71 -4
@@ -632,6 +632,21 @@ def fgetpath(fd, fh = None, mountpoint = None): break return ret +# Filesystem properties + +FSPROP_NAMESPACE = "trusted." +FSPROP_NAME_PREFIX = "xfs:" +FSPROP_AUTOFSCK_NAME = "autofsck" + +def fsprop_attrname(n): + '''Construct the xattr name for a filesystem property.''' + return f"{FSPROP_NAMESPACE}{FSPROP_NAME_PREFIX}{n}" + +def fsprop_getstr(fd, n): + '''Return the value of a filesystem property as a string.''' + attrname = fsprop_attrname(n) + return os.getxattr(fd, attrname).decode('utf-8') + # main program def health_reports(mon_fp):
@@ -869,6 +884,31 @@ def check_monitor(mountpoint, fd): # Monitor available; success! return 0 +def want_repair_from_autofsck(fd): + '''Determine want_repair from the autofsck filesystem property.''' + global has_parent + global has_rmapbt + + try: + advice = fsprop_getstr(fd, FSPROP_AUTOFSCK_NAME) + if advice == "repair": + return True + if advice == "check" or advice == "optimize": + return False + if advice == "none": + return None + except: + # Any OS error (including ENODATA) or string parsing error is + # treated the same as an unrecognized value. + pass + + # For an unrecognized value, log but do not fix runtime corruption if + # backref metadata are enabled. If no backref metadata are available, + # the fs is too old so don't run at all. + if has_rmapbt or has_parent: + return False + return None + def monitor(mountpoint, event_queue, check, **kwargs): '''Monitor the given mountpoint for health events.''' global everything
@@ -877,6 +917,7 @@ def monitor(mountpoint, event_queue, check, **kwargs): global want_repair global has_parent global has_rmapbt + use_autofsck = want_repair is None def event_loop(mon_fd, event_queue, fh): # Ownership of mon_fd (and hence responsibility for closing it)
@@ -908,12 +949,33 @@ def monitor(mountpoint, event_queue, check, **kwargs): msg = _("detecting fs features") eprintln(_(f'{printf_prefix}: {msg}: {e}')) + # Does the sysadmin have any advice for us about whether or not to + # background scrub? + if use_autofsck: + want_repair = want_repair_from_autofsck(fd) + if want_repair is None: + msg = _("Disabling daemon per autofsck directive.") + printlogln(f"{mountpoint}: {msg}") + os.close(fd) + return 0 + elif want_repair: + msg = _("Automatically repairing per autofsck directive.") + printlogln(f"{mountpoint}: {msg}") + else: + msg = _("Only logging errors per autofsck directive.") + printlogln(f"{mountpoint}: {msg}") + # Check that the kernel supports repairs at all. if want_repair and not xfs_repair_is_supported(fd): - msg = _("XFS online repair is not supported, exiting") + if not use_autofsck: + msg = _("XFS online repair is not supported, exiting") + printlogln(f"{mountpoint}: {msg}") + os.close(fd) + return 1 + + msg = _("XFS online repair is not supported, will report only") printlogln(f"{mountpoint}: {msg}") - os.close(fd) - return 1 + want_repair = False # Check for the backref metadata that makes repair effective. if want_repair:
@@ -1114,8 +1176,11 @@ def main(): parser.add_argument('--event-schema', type = str, \ default = '@pkg_data_dir@/xfs_healthmon.schema.json', \ help = argparse.SUPPRESS) - parser.add_argument("--repair", action = "store_true", \ + action_group = parser.add_mutually_exclusive_group() + action_group.add_argument("--repair", action = "store_true", \ help = _("Always repair corrupt metadata")) + action_group.add_argument("--autofsck", action = "store_true", \ + help = _("Use the \"autofsck\" fs property to decide to repair")) parser.add_argument("--check", action = "store_true", \ help = _("Check that health monitoring is supported")) args = parser.parse_args()
@@ -1148,6 +1213,8 @@ def main(): everything = True if args.debug_fast: debug_fast = True + if args.autofsck: + want_repair = None if args.repair: want_repair = True
healer/[email protected] +1 -1
@@ -18,7 +18,7 @@ RequiresMountsFor=%f Type=exec Environment=SERVICE_MODE=1 ExecCondition=@pkg_libexec_dir@/xfs_healer --check %f -ExecStart=@pkg_libexec_dir@/xfs_healer --log %f +ExecStart=@pkg_libexec_dir@/xfs_healer --autofsck --log %f SyslogIdentifier=%N # Create the service underneath the healer background service slice so that we
PATCH 21/26 xfs_healer: run full scrub after lost corruption events or targeted repair failure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If we fail to perform a spot repair of metadata or the kernel tells us that it lost corruption events due to queue limits, initiate a full run of the online fsck service to try to fix the error.
--- healer/Makefile | 1 + healer/xfs_healer.py.in | 59 +++++++++++++++++++++++++++++++++++++++++------ include/builddefs.in | 1 + scrub/Makefile | 7 ++---- 4 files changed, 57 insertions(+), 11 deletions(-)
healer/Makefile +1 -0
@@ -34,6 +34,7 @@ $(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext -e '/@INIT_GETTEXT@/r $(TOPDIR)/libfrog/gettext.py' \ -e '/@INIT_GETTEXT@/d' \ -e "s|@pkg_data_dir@|$(PKG_DATA_DIR)|g" \ + -e "s|@scrub_svcname@|$(XFS_SCRUB_SVCNAME)|g" \ < $< > $@ $(Q)chmod a+x $@
healer/xfs_healer.py.in +52 -7
135 lines not on this page · raw message
include/builddefs.in +1 -0
@@ -62,6 +62,7 @@ MKFS_CFG_DIR = @datadir@/@pkg_name@/mkfs PKG_STATE_DIR = @localstatedir@/lib/@pkg_name@ XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP=$(PKG_STATE_DIR)/xfs_scrub_all_media.stamp [email protected] CC = @cc@ BUILD_CC = @BUILD_CC@
scrub/Makefile +3 -4
@@ -8,7 +8,6 @@ include $(builddefs) SCRUB_PREREQS=$(HAVE_GETFSMAP) [email protected] [email protected] ifeq ($(SCRUB_PREREQS),yes)
@@ -21,7 +20,7 @@ XFS_SCRUB_SERVICE_ARGS = -b -o autofsck ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\ - $(scrub_svcname) \ + $(XFS_SCRUB_SVCNAME) \ [email protected] \ $(scrub_media_svcname) \ [email protected] \
@@ -123,7 +122,7 @@ xfs_scrub_all.timer: xfs_scrub_all.timer.in $(builddefs) $(XFS_SCRUB_ALL_PROG): $(XFS_SCRUB_ALL_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext.py @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ - -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@scrub_svcname@|$(XFS_SCRUB_SVCNAME)|g" \ -e "s|@scrub_media_svcname@|$(scrub_media_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@stampfile@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP)|g" \
@@ -137,7 +136,7 @@ $(XFS_SCRUB_ALL_PROG): $(XFS_SCRUB_ALL_PROG).in $(builddefs) $(TOPDIR)/libfrog/g xfs_scrub_fail: xfs_scrub_fail.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ - -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@scrub_svcname@|$(XFS_SCRUB_SVCNAME)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" < $< > $@ $(Q)chmod a+x $@
PATCH 22/26 xfs_healer: use getmntent to find moved filesystems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Wrap the libc getmntent function in an iterator. This enables xfs_healer to record the fsname (or fs spec) of the mountpoint that it's running against, and use that fsname to walk /proc/mounts to re-find the filesystem if the mount has moved elsewhere if it needs to open the fs to perform repairs.
--- healer/xfs_healer.py.in | 117 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 114 insertions(+), 3 deletions(-)
healer/xfs_healer.py.in +114 -3
@@ -80,6 +80,7 @@ want_repair = False has_parent = False has_rmapbt = False libhandle = None +libc = None def printlogln(*args, **kwargs): '''Print a log message to stdout and flush it.'''
@@ -184,6 +185,16 @@ class xfs_handle(ctypes.Structure): ] assert ctypes.sizeof(xfs_handle) == 24 +def find_xfs_dev(mountpoint): + '''Find the xfs device for a particular mount point or raise exception.''' + + for mnt in some_mount_entries( + lambda mnt: mnt.type == 'xfs' and \ + mnt.dir == mountpoint): + return mnt.fsname + + raise Exception(_('Cannot find xfs device')) + class fshandle(object): def __init__(self, fd, mountpoint = None): global libhandle
@@ -193,6 +204,7 @@ class fshandle(object): if isinstance(fd, fshandle): # copy an existing fshandle + self.fsname = fd.fsname self.mountpoint = fd.mountpoint ctypes.pointer(self.handle)[0] = fd.handle return
@@ -201,6 +213,7 @@ class fshandle(object): raise Exception(_('fshandle needs a mountpoint')) self.mountpoint = mountpoint + self.fsname = find_xfs_dev(mountpoint) # Create the file and fs handles for the open mountpoint # so that we can compare them later
@@ -222,15 +235,16 @@ class fshandle(object): hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) self.handle = hanp.contents - def reopen(self): - '''Reopen a file handle obtained via weak reference.''' + def reopen_from(self, mountpoint): + '''Reopen a file handle obtained via weak reference, using + a specific mount point.''' global libhandle global printf_prefix buf = ctypes.c_void_p() buflen = ctypes.c_size_t() - fd = os.open(self.mountpoint, os.O_RDONLY) + fd = os.open(mountpoint, os.O_RDONLY) # Create the file and fs handles for the open mountpoint # so that we can compare them later
@@ -258,6 +272,26 @@ class fshandle(object): libhandle.free_handle(buf, buflen) return fd + def reopen(self): + '''Reopen a file handle obtained via weak reference.''' + + # First try the original mountpoint + try: + return self.reopen_from(self.mountpoint) + except Exception as e: + # Now scan /proc/self/mounts for any other bind mounts + # of this filesystem + for mnt in some_mount_entries( + lambda mnt: mnt.type == 'xfs' and \ + mnt.fsname == self.fsname): + try: + return self.reopen_from(mnt.dir) + except: + pass + + # Return original error + raise e + def subst(self, ino, gen): '''Substitute the inode and generation components of a handle.''' self.handle.ha_fid.fid_ino = ino
@@ -302,6 +336,77 @@ def libhandle_load(): ctypes.c_void_p, ctypes.c_size_t) +class libc_mntent(ctypes.Structure): + _fields_ = [ + ("mnt_fsname", ctypes.c_char_p), + ("mnt_dir", ctypes.c_char_p), + ("mnt_type", ctypes.c_char_p), + ("mnt_opts", ctypes.c_char_p), + ("mnt_freq", ctypes.c_int), + ("mnt_passno", ctypes.c_int), + ] + +class MountEntry(object): + '''Description of a mounted filesystem.''' + def __init__(self, fsname, dir, type, opts): + self.fsname = fsname + self.dir = pathlib.Path(dir.decode('utf-8')) + self.type = type.decode('utf-8') + self.opts = opts + +def mount_entries(): + '''Iterate all mounted filesystems in the system.''' + try: + fp = libc.setmntent(b"/proc/self/mounts", b"r") + mntbuf = libc_mntent() + namebuflen = 4096 + namebuf = ctypes.create_string_buffer(namebuflen) + mnt = libc.getmntent_r(fp, mntbuf, namebuf, namebuflen); + while mnt: + yield MountEntry(mnt.contents.mnt_fsname, + mnt.contents.mnt_dir, + mnt.contents.mnt_type, + mnt.contents.mnt_opts) + mnt = libc.getmntent_r(fp, mntbuf, namebuf, namebuflen); + finally: + libc.endmntent(fp); + +def some_mount_entries(filter_fn): + '''Iterate some of the mounted filesystems in the system.''' + return filter(filter_fn, mount_entries()) + +def libc_load(): + '''Load libc and set things up.''' + global libc + + soname = ctypes.util.find_library('c') + if soname is None: + errstr = os.strerror(errno.ENOENT) + msg = _("while finding library") + raise OSError(errno.ENOENT, f'{msg}: {errstr}', 'libc') + + libc = ctypes.CDLL(soname, use_errno = True) + + libc.setmntent.argtypes = ( + ctypes.c_char_p, + ctypes.c_char_p) + libc.setmntent.restype = ctypes.c_void_p + + libc.getmntent_r.argtypes = ( + ctypes.c_void_p, + ctypes.POINTER(libc_mntent), + ctypes.POINTER(ctypes.c_char), + ctypes.c_int) + libc.getmntent_r.restype = ctypes.POINTER(libc_mntent) + + libc.getmntent.argtypes = ( + ctypes.c_void_p,) + libc.getmntent.restype = ctypes.POINTER(libc_mntent) + + libc.endmntent.argtypes = ( + ctypes.c_void_p,) + libc.endmntent.restype = ctypes.c_int + # metadata scrubbing stuff XFS_SCRUB_TYPE_PROBE = 0 XFS_SCRUB_TYPE_SB = 1
@@ -1244,6 +1349,12 @@ def main(): if not validator_fn: return 1 + try: + libc_load() + except OSError as e: + eprintln(f"libc: {e}") + return 1 + try: libhandle_load() except OSError as e:
PATCH 23/26 xfs_healer: validate that repair fds point to the monitored fs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When xfs_healer reopens a mountpoint to perform a repair, it should validate that the opened fd points to a file on the same filesystem as the one being monitored.
--- healer/xfs_healer.py.in | 53 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 12 deletions(-)
healer/xfs_healer.py.in +41 -12
@@ -163,6 +163,28 @@ def open_health_monitor(fd, verbose = False): ret = fcntl.ioctl(fd, XFS_IOC_HEALTH_MONITOR, arg) return ret +class xfs_health_samefs(ctypes.Structure): + _fields_ = [ + ('fd', ctypes.c_int), + ('flags', ctypes.c_uint), + ] +assert ctypes.sizeof(xfs_health_samefs) == 8 + +XFS_IOC_HEALTH_SAMEFS = _IOW(0x58, 69, xfs_health_samefs) + +def is_same_fs(mon_fd, fd): + '''Does fd point to the same filesystem as the monitor fd?''' + arg = xfs_health_samefs() + arg.fd = fd + arg.flags = 0 + + try: + fcntl.ioctl(mon_fd, XFS_IOC_HEALTH_SAMEFS, arg) + except OSError as e: + # any error means this isn't the right fs mount + return False + return True + # libhandle stuff class xfs_fsid(ctypes.Structure): _fields_ = [
@@ -235,7 +257,7 @@ class fshandle(object): hanp = ctypes.cast(buf, ctypes.POINTER(xfs_handle)) self.handle = hanp.contents - def reopen_from(self, mountpoint): + def reopen_from(self, mountpoint, is_acceptable): '''Reopen a file handle obtained via weak reference, using a specific mount point.''' global libhandle
@@ -261,7 +283,8 @@ class fshandle(object): # Did we get the same handle? if buflen.value != ctypes.sizeof(xfs_handle) or \ - bytes(hanp.contents) != bytes(self.handle): + bytes(hanp.contents) != bytes(self.handle) or \ + not is_acceptable(fd): os.close(fd) libhandle.free_handle(buf, buflen) msg = _("reopening")
@@ -270,14 +293,15 @@ class fshandle(object): printf_prefix) libhandle.free_handle(buf, buflen) + return fd - def reopen(self): + def reopen(self, is_acceptable = lambda x: True): '''Reopen a file handle obtained via weak reference.''' # First try the original mountpoint try: - return self.reopen_from(self.mountpoint) + return self.reopen_from(self.mountpoint, is_acceptable) except Exception as e: # Now scan /proc/self/mounts for any other bind mounts # of this filesystem
@@ -285,7 +309,8 @@ class fshandle(object): lambda mnt: mnt.type == 'xfs' and \ mnt.fsname == self.fsname): try: - return self.reopen_from(mnt.dir) + return self.reopen_from(mnt.dir, + is_acceptable) except: pass
@@ -893,7 +918,7 @@ def report_shutdown(event): msg = _("filesystem shut down due to") printlogln(f"{printf_prefix}: {msg} {some_reasons}") -def handle_event(lines, fh, everything): +def handle_event(lines, fh, everything, mon_fd): '''Handle an event asynchronously.''' global log global has_parent
@@ -981,7 +1006,7 @@ def handle_event(lines, fh, everything): if maybe_pathify: pathify_event(event, fh) - repair_metadata(event, fh) + repair_metadata(event, fh, mon_fd) def check_monitor(mountpoint, fd): '''Check if the kernel can send us health events for the given mountpoint.'''
@@ -1045,7 +1070,7 @@ def monitor(mountpoint, event_queue, check, **kwargs): nr = 0 for lines in health_reports(mon_fp): event_queue.submit(handle_event, lines, fh, - everything) + everything, mon_fd) # Periodically run the garbage collector to # constrain memory usage in the main thread.
@@ -1055,6 +1080,12 @@ def monitor(mountpoint, event_queue, check, **kwargs): gc.collect() nr += 1 + # Once we run out of events to process, shut down all + # the workers and wait for them to complete before we + # close mon_fp so that repair reopen can't walk off + # freed mon_fd. + event_queue.shutdown(wait = True, cancel_futures = True) + try: fd = os.open(mountpoint, os.O_RDONLY) except Exception as e:
@@ -1244,7 +1275,7 @@ def repair_inode(event, fd): except Exception as e: eprintln(f"{prefix} {structure}: {e}") -def repair_metadata(event, fh): +def repair_metadata(event, fh, mon_fd): '''Repair a metadata corruption.''' global debug global printf_prefix
@@ -1253,7 +1284,7 @@ def repair_metadata(event, fh): printlogln(f'repair {event}') try: - fd = fh.reopen() + fd = fh.reopen(lambda fdx: is_same_fs(mon_fd, fdx)) except Exception as e: eprintln(f"{printf_prefix}: {e}") return
@@ -1388,8 +1419,6 @@ def main(): # Consider SIGINT to be a clean exit. pass - args.event_queue.shutdown() - # See the service mode comments in xfs_scrub.c for why we sleep and # compress all nonzero exit codes to 1. if 'SERVICE_MODE' in os.environ:
PATCH 24/26 xfs_healer: add a manual page
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new section 8 manpage for this service daemon so others can read about what this program is supposed to do.
--- man/man8/xfs_healer.8 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 man/man8/xfs_healer.8
man/man8/xfs_healer.8 new file +85 -0
@@ -0,0 +1,85 @@ +.TH xfs_healer 8 +.SH NAME +xfs_healer \- automatically heal damage to XFS filesystem metadata +.SH SYNOPSIS +.B xfs_healer +[ +.B OPTIONS +] +.I mount-point +.br +.B xfs_healer \-V +.SH DESCRIPTION +.B xfs_healer +is a daemon that tries to automatically repair damaged XFS filesystem metadata. +.PP +.B WARNING! +This program is +.BR EXPERIMENTAL "," +which means that its behavior and interface +could change at any time! +.PP +.B xfs_healer +asks the kernel to report all observations of corrupt metadata, media errors, +filesystem shutdowns, and file I/O errors. +The program can respond to runtime metadata corruption errors by initiating +targeted repairs of the suspect metadata or a full online fsck of the +filesystem. + +Normally this program runs as a systemd service. +The service is activated through udev whenever a filesystem is mounted. +Only systemd is supported. + +The kernel may not support repairing or optimizing the filesystem. +If this is the case, the filesystem must be unmounted and +.BR xfs_repair (8) +run on the filesystem to fix the problems. +.SH OPTIONS +.TP +.B \-\-autofsck +Use the +.I autofsck +filesystem property to decide whether or not to repair corrupt metadata. +See the +.B \-\-repair +option for more details. +If this option is specified but the kernel does not support repairs, the +program will report but not act upon corruptions. +.TP +.B \-\-check +Check if the filesystem supports sending health events. +Exits with 0 if it does, and non-zero if not. +.TP +.BI \-\-everything +Ask the kernel to send us good metadata health events, not only events related +to metadata corruption, media errors, shutdowns, and I/O errors. +.TP +.B \-\-log +Print every event to standard output. +.TP +.B \-\-repair +Always try to repair each piece of corrupt metadata when the kernel tells us +about it. +If an individual repair fails or the kernel tells us that health events were +lost, the +.I xfs_scrub +service for this mount point will be launched. +The default is not to try to repair anything. +If this option is specified but the kernel does not support repairs, the +program will exit. +.TP +.BI \-V +Prints the version number and exit. +.SH CAVEATS +.B xfs_healer +is an immature utility! +Do not run this program unless you have backups of your data! +This program takes advantage of in-kernel scrubbing to verify a given +data structure with locks held and can keep the filesystem busy for a +long time. +The kernel must be new enough to support the SCRUB_METADATA ioctl. +.PP +If errors are found and cannot be repaired, the filesystem must be +unmounted and repaired. +.SH SEE ALSO +.BR xfs_repair (8).
PATCH 25/26 xfs_scrub: report media scrub failures to the kernel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the media scan finds that media have been lost, report this to the kernel so that the healthmon code can pass that along to xfs_healer.
--- scrub/phase6.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
scrub/phase6.c +25 -0
@@ -673,6 +673,29 @@ clean_pool(return ret; } +static void +report_ioerr_to_kernel ( +struct scrub_ctx *ctx, +struct disk *disk, +uint64_t start, +uint64_t length) +{ +struct xfs_media_error me = { + .daddr = start, + .bbcount = length, + }; +dev_t dev =disk_to_dev (ctx, disk); + +if (dev == ctx->fsinfo.fs_datadev) + me.flags |=XFS_MEDIA_ERROR_DATADEV ; +else if (dev == ctx->fsinfo.fs_rtdev) + me.flags |=XFS_MEDIA_ERROR_RTDEV ; +else if (dev == ctx->fsinfo.fs_logdev) + me.flags |=XFS_MEDIA_ERROR_LOGDEV ; + +ioctl (ctx->mnt.fd,XFS_IOC_MEDIA_ERROR , &me); +} +/* Remember a media error for later. */ static void remember_ioerr(
@@ -697,6 +720,8 @@ remember_ioerr(return ; } +report_ioerr_to_kernel (ctx, disk, start, length); + tree =bitmap_for_disk (ctx, disk, vs);if (!tree) {str_liberror (ctx,ENOENT ,_ ("finding bad block bitmap" ));
PATCH 26/26 debian: enable xfs_healer on the root filesystem by default
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we're finished building autonomous repair, enable the service on the root filesystem by default. The root filesystem is mounted by the initrd prior to starting systemd, which is why the udev rule cannot autostart the service for the root filesystem.
dh_installsystemd won't activate a template service (aka one with an at-sign in the name) even if it provides a DefaultInstance directive to make that possible. Use a fugly shim for this.
--- debian/control | 2 +- debian/postinst | 8 ++++++++ debian/prerm | 13 +++++++++++++ debian/rules | 2 +- 4 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 debian/prerm
debian/control +1 -1
@@ -10,7 +10,7 @@ Homepage: https://xfs.wiki.kernel.org/ Package: xfsprogs Depends: ${shlibs:Depends}, ${misc:Depends}, python3-dbus, python3:any Provides: fsck-backend -Suggests: xfsdump, acl, attr, quota +Suggests: xfsdump, acl, attr, quota, python3-jsonschema Breaks: xfsdump (<< 3.0.0) Replaces: xfsdump (<< 3.0.0) Architecture: linux-any
debian/postinst +8 -0
@@ -24,5 +24,13 @@ case "${1}" in esac #DEBHELPER# +# +# dh_installsystemd doesn't handle template services even if we supply a +# default instance, so we'll install it here. +if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ] ; then + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + /bin/systemctl enable [email protected] || true + fi +fi exit 0
debian/prerm new file +13 -0
@@ -0,0 +1,13 @@ +#!/bin/sh + +set -e + +# dh_installsystemd doesn't handle template services even if we supply a +# default instance, so we'll install it here. +if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = remove ] && [ -d /run/systemd/system ] ; then + /bin/systemctl disable [email protected] || true +fi + +#DEBHELPER# + +exit 0
debian/rules +1 -1
@@ -97,4 +97,4 @@ override_dh_installdocs: dh_installdocs -XCHANGES override_dh_installsystemd: - dh_installsystemd -p xfsprogs --no-restart-after-upgrade --no-stop-on-upgrade system-xfs_scrub.slice xfs_scrub_all.timer + dh_installsystemd -p xfsprogs --no-restart-after-upgrade --no-stop-on-upgrade system-xfs_scrub.slice xfs_scrub_all.timer system-xfs_healer.slice
PATCH v2 2/2 xfsprogs: autonomous self healing of filesystems in Rust
In reply to Darrick J. WongHi all,
The initial implementation of the self healing daemon is written in Python. This was useful for rapid prototyping, but a more performant and typechecked codebase is valuable. Write a second implementation in Rust to reduce event processing overhead and library dependence. This could have been done in C, but I decided to use an environment with somewhat fewer footguns.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring-rust
---
Commits in this patchset:
* xfs_healer: start building a Rust version
* xfs_healer: enable gettext for localization
* xfs_healer: bindgen xfs_fs.h
* xfs_healer: define Rust objects for health events and kernel interface
* xfs_healer: read binary health events from the kernel
* xfs_healer: read json health events from the kernel
* xfs_healer: create a weak file handle so we don't pin the mount
* xfs_healer: fix broken filesystem metadata
* xfs_healer: check for fs features needed for effective repairs
* xfs_healer: use getparents to look up file names
* xfs_healer: make the rust program check if kernel support available
* xfs_healer: use the autofsck fsproperty to select mode
* xfs_healer: use rc on the mountpoint instead of lifetime annotations
* xfs_healer: use thread pools
* xfs_healer: run full scrub after lost corruption events or targeted repair failure
* xfs_healer: use getmntent in Rust to find moved filesystems
* xfs_healer: validate that repair fds point to the monitored fs in Rust
* debian/control: listify the build dependencies
* debian/control: pull in build dependencies for xfs_healer
--- healer/bindgen_xfs_fs.h | 6 + configure.ac | 84 ++++++++ debian/control | 30 +++ debian/rules | 3 healer/.cargo/config.toml.system | 6 + healer/Cargo.toml.in | 37 +++ healer/Makefile | 143 +++++++++++++ healer/rbindgen | 57 +++++ healer/src/fsgeom.rs | 41 ++++ healer/src/fsprops.rs | 101 +++++++++ healer/src/getmntent.rs | 117 +++++++++++ healer/src/getparents.rs | 210 ++++++++++++++++++++ healer/src/healthmon/cstruct.rs | 354 +++++++++++++++++++++++++++++++++ healer/src/healthmon/event.rs | 122 +++++++++++ healer/src/healthmon/fs.rs | 163 +++++++++++++++ healer/src/healthmon/groups.rs | 160 +++++++++++++++ healer/src/healthmon/inodes.rs | 142 +++++++++++++ healer/src/healthmon/json.rs | 409 ++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 47 ++++ healer/src/healthmon/samefs.rs | 33 +++ healer/src/lib.rs | 17 ++ healer/src/main.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/repair.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/util.rs | 81 ++++++++ healer/src/weakhandle.rs | 209 +++++++++++++++++++ healer/src/xfs_types.rs | 292 +++++++++++++++++++++++++++ healer/src/xfsprogs.rs.in | 33 +++ include/builddefs.in | 13 + include/buildrules | 1 m4/Makefile | 1 m4/package_rust.m4 | 163 +++++++++++++++ 31 files changed, 3851 insertions(+), 4 deletions(-) create mode 100644 healer/bindgen_xfs_fs.h create mode 100644 healer/.cargo/config.toml.system create mode 100644 healer/Cargo.toml.in create mode 100755 healer/rbindgen create mode 100644 healer/src/fsgeom.rs create mode 100644 healer/src/fsprops.rs create mode 100644 healer/src/getmntent.rs create mode 100644 healer/src/getparents.rs create mode 100644 healer/src/healthmon/cstruct.rs create mode 100644 healer/src/healthmon/event.rs create mode 100644 healer/src/healthmon/fs.rs create mode 100644 healer/src/healthmon/groups.rs create mode 100644 healer/src/healthmon/inodes.rs create mode 100644 healer/src/healthmon/json.rs create mode 100644 healer/src/healthmon/mod.rs create mode 100644 healer/src/healthmon/samefs.rs create mode 100644 healer/src/lib.rs create mode 100644 healer/src/main.rs create mode 100644 healer/src/repair.rs create mode 100644 healer/src/util.rs create mode 100644 healer/src/weakhandle.rs create mode 100644 healer/src/xfs_types.rs create mode 100644 healer/src/xfsprogs.rs.in create mode 100644 m4/package_rust.m4
PATCH 1/19 xfs_healer: start building a Rust version
From: Darrick J. Wong <[email protected]>
Update the build infrastructure to support building rust programs and add a stub xfs_healer program. This is a little gross because cargo install doesn't handle packaging, according to its maintainers.
--- configure.ac | 59 ++++++++++++++++++ debian/rules | 2 + healer/.cargo/config.toml.system | 6 ++ healer/Cargo.toml.in | 15 ++++ healer/Makefile | 96 ++++++++++++++++++++++++++++- healer/src/lib.rs | 7 ++ healer/src/main.rs | 110 +++++++++++++++++++++++++++++++++ healer/src/xfsprogs.rs.in | 12 ++++ include/builddefs.in | 9 +++ m4/Makefile | 1 m4/package_rust.m4 | 128 ++++++++++++++++++++++++++++++++++++++ 11 files changed, 443 insertions(+), 2 deletions(-) create mode 100644 healer/.cargo/config.toml.system create mode 100644 healer/Cargo.toml.in create mode 100644 healer/src/lib.rs create mode 100644 healer/src/main.rs create mode 100644 healer/src/xfsprogs.rs.in create mode 100644 m4/package_rust.m4
configure.ac +59 -0
@@ -13,6 +13,10 @@ fi if test "${CXXFLAGS+set}" != "set"; then CXXFLAGS="-g -O2 -std=gnu++11" fi +if test "${CARGOFLAGS+set}" != "set"; then + CARGOFLAGS="" +fi +AC_SUBST(CARGOFLAGS) AC_PROG_INSTALL LT_INIT
@@ -116,6 +120,29 @@ AC_ARG_ENABLE(healer, enable_healer=yes) AC_SUBST(enable_healer) +# Is this a release build? Mostly important for cargo/rustc. +AC_ARG_ENABLE(release, +[ --enable-release=[yes/no] This is a release build [[default=no]]],, + enable_release=no) +AC_SUBST(enable_release) + +# Should we build Rust with the system crates? "yes" means it's required, +# "no" means use crates.io, "probe" means figure it out from the distro. +AC_ARG_WITH([system-crates], + [AS_HELP_STRING([--with-system-crates=[yes/no/probe]], +[Build Rust programs with system crates instead of downloading from crates.io. [default=no]])], + [], + [with_system_crates=no]) +AC_SUBST(with_system_crates) + +# Should we check for Rust crates and skip builds if they are not installed? +# Distros that package crates themselves and establish build dependencies on +# those packages can skip the checks. +AC_ARG_ENABLE(crate-checks, +[ --enable-crate-checks=[yes/no] Check for Rust crates before building [[default=yes]]],, + enable_crate_checks=yes) +AC_SUBST(enable_crate_checks) + # # If the user specified a libdir ending in lib64 do not append another # 64 to the library names.
@@ -224,5 +251,37 @@ fi AC_MANUAL_FORMAT AC_HAVE_LIBURCU_ATOMIC64 +# Check for a Rust compiler +# XXX: I don't know how to cross compile Rust yet +if test "$host" = "$build"; then + AC_HAVE_RUSTC +fi + +# If we have rustc, check if LTO is supported (it should be) +if test "$have_rustc" = "yes"; then + if test "$enable_lto" = "yes" || test "$enable_lto" = "probe"; then + AC_RUSTC_CHECK_LTO + fi +fi +if test "$enable_lto" = "yes" && test "$have_rustc_lto" != "yes"; then + AC_MSG_ERROR([LTO not supported by Rust compiler.]) +fi + +# If we still have rustc, check that we have cargo for crate management +if test "$have_rustc" = "yes"; then + AC_HAVE_CARGO +fi + +# If we have cargo, check that our crate dependencies are present +if test "$have_cargo" = "yes"; then + if test "$with_system_crates" = "yes"; then + AC_USE_SYSTEM_CRATES + elif test "$with_system_crates" = "probe"; then + AC_MAYBE_USE_SYSTEM_CRATES + fi + AC_HAVE_CLIPPY + AC_HAVE_HEALER_CRATES +fi + AC_CONFIG_FILES([include/builddefs]) AC_OUTPUT
debian/rules +2 -0
@@ -39,6 +39,8 @@ configure_options = \ --disable-addrsan \ --disable-threadsan \ --enable-lto \ + --enable-release \ + --with-system-crates \ --localstatedir=/var options = export DEBUG=-DNDEBUG DISTRIBUTION=debian \
healer/.cargo/config.toml.system new file +6 -0
@@ -0,0 +1,6 @@ +# XXX gross hack so that we don't download crates from the internet +[source] +[source.system-packages] +directory = "/usr/share/cargo/registry" +[source.crates-io] +replace-with = "system-packages"
healer/Cargo.toml.in new file +15 -0
@@ -0,0 +1,15 @@ +[package] +name = "xfs_healer" +version = "@pkg_version@" +edition = "2021" + +[profile.dev] +lto = @cargo_lto@ + +[profile.release] +lto = @cargo_lto@ + +# Be sure to update AC_HAVE_HEALER_CRATES if you update the dependency list. +[dependencies] +clap = { version = "4.0.32", features = ["derive"] } +anyhow = { version = "1.0.69" }
healer/Makefile +94 -2
124 lines not on this page · raw message
healer/src/lib.rs new file +7 -0
@@ -0,0 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright (C) 2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ + +pub mod xfsprogs;
healer/src/main.rs new file +110 -0
110 lines not on this page · raw message
healer/src/xfsprogs.rs.in new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2025 Oracle. All Rights Reserved. + * Author: Darrick J. Wong <[email protected]> + */ +pub const VERSION: &str = "@pkg_version@"; + +/// Dummy function to simulate looking up a string in a message catalog +#[allow(non_snake_case)] +pub fn M_<T: Into<String>>(msgid: T) -> String { + msgid.into() +}
include/builddefs.in +9 -0
@@ -17,6 +17,15 @@ CFLAGS = @CFLAGS@ -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 -Wno-address-of-packed- CXXFLAGS = @CXXFLAGS@ -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 -Wno-address-of-packed-member BUILD_CFLAGS = @BUILD_CFLAGS@ -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 +ENABLE_RELEASE = @enable_release@ +HAVE_RUSTC = @have_rustc@ +HAVE_RUSTC_LTO = @have_rustc_lto@ +HAVE_CARGO = @have_cargo@ +HAVE_HEALER_CRATES = @have_healer_crates@ +CARGOFLAGS = @CARGOFLAGS@ +USE_SYSTEM_CRATES = @use_system_crates@ +HAVE_CLIPPY = @have_clippy@ + # make sure we don't pick up whacky LDFLAGS from the make environment and # only use what we calculate from the configured options above. LDFLAGS =
m4/Makefile +1 -0
@@ -23,6 +23,7 @@ LSRCFILES = \ package_sanitizer.m4 \ package_services.m4 \ package_icu.m4 \ + package_rust.m4 \ package_urcu.m4 \ package_utilies.m4 \ package_uuiddev.m4 \
m4/package_rust.m4 new file +128 -0
128 lines not on this page · raw message
PATCH 2/19 xfs_healer: enable gettext for localization
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Include gettext-rs in our Rust application if the builder wants localization. It's not clear to me what we're really supposed to use to localize Rust programs, but xfsprogs uses gettext so let's just plug into that for now. Note that xgettext prior to 0.24 doesn't technically support Rust, but it matches patterns well enough to extract simple format strings (e.g. M_("hello")) despite the warnings.
--- configure.ac | 13 ++++++++++++- healer/Cargo.toml.in | 15 +++++++++++++++ healer/Makefile | 16 ++++++++++++++++ healer/src/main.rs | 4 +++- healer/src/xfsprogs.rs.in | 20 ++++++++++++++++++++ include/builddefs.in | 1 + include/buildrules | 1 + m4/package_rust.m4 | 6 ++++++ 8 files changed, 74 insertions(+), 2 deletions(-)
configure.ac +12 -1
@@ -143,6 +143,17 @@ AC_ARG_ENABLE(crate-checks, enable_crate_checks=yes) AC_SUBST(enable_crate_checks) +# Some distributions do not package gettext-rs; provide a way to disable it +# for Rust without disabling it for the C programs. +AC_ARG_ENABLE(gettext-rs, +[ --enable-gettext-rs=[yes/no] Enable gettext-rs support if gettext is enabled],, + enable_gettext_rs="$enable_gettext") +# If the main gettext is not enabled, then we don't want the Rust version. +if test "$enable_gettext" = "no"; then + enable_gettext_rs="no" +fi +AC_SUBST(enable_gettext_rs) + # # If the user specified a libdir ending in lib64 do not append another # 64 to the library names.
@@ -163,7 +174,7 @@ test -n "$multiarch" && enable_lib64=no # to "find" is required, to avoid including such directories in the # list. LOCALIZED_FILES="" -for lfile in `find ${srcdir} -path './.??*' -prune -o -name '*.c' -print -o -name '*.py.in' -print || exit 1`; do +for lfile in `find ${srcdir} -path './.??*' -prune -o -name '*.c' -print -o -name '*.py.in' -print -o -name '*.rs' -print || exit 1`; do LOCALIZED_FILES="$LOCALIZED_FILES \$(TOPDIR)/$lfile" done AC_SUBST(LOCALIZED_FILES)
healer/Cargo.toml.in +15 -0
@@ -9,7 +9,22 @@ lto = @cargo_lto@ [profile.release] lto = @cargo_lto@ +[features] +@cargo_cmt_gettext@gettext = ["dep:gettext-rs"] + # Be sure to update AC_HAVE_HEALER_CRATES if you update the dependency list. [dependencies] clap = { version = "4.0.32", features = ["derive"] } anyhow = { version = "1.0.69" } + +# XXX: Crates with major version 0 are not considered ABI-stable, so the minor +# version is treated as if it were the major version. This creates problems +# pulling in distro-packaged crates, so we only specify the dependency as major +# version 0. Until these crates reach 1.0.0, we'll have to patch when things +# break. Ref: +# https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html + +# Dynamically comment out all the gettextrs related dependency information in +# Cargo.toml becuse cargo requires the crate to be present so that it can +# generate a Cargo.lock file even if the build never uses it. +@cargo_cmt_gettext@gettext-rs = { version = "0", optional = true } # 0.7.0
healer/Makefile +16 -0
@@ -44,6 +44,19 @@ else CARGO_INSTALL_FLAGS=--debug endif +# Enable gettext if it's available +ifeq ($(ENABLE_GETTEXT_RS),yes) +CARGO_BUILD_FLAGS+=--features gettext +CARGO_CLIPPY_FLAGS+=--features gettext +CARGO_INSTALL_FLAGS+=--features gettext +CARGO_CMT_GETTEXT= +else + # This is what you have to do to define a variable to a octothorpe. + define CARGO_CMT_GETTEXT +# +endef +endif + # Assume that if rustc supports LTO then cargo knows how to configure it. # rustc and cargo support LTO as of Rust-2021. ifeq ($(HAVE_RUSTC_LTO),yes)
@@ -97,12 +110,15 @@ $(XFS_HEALER_PROG): $(XFS_HEALER_PROG).in $(builddefs) $(TOPDIR)/libfrog/gettext src/xfsprogs.rs: src/xfsprogs.rs.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@PACKAGE@|$(PKG_NAME)|g" \ + -e "s|@LOCALEDIR@|$(PKG_LOCALE_DIR)|g" \ < $< > $@ $(CARGO_MANIFEST): $(CARGO_MANIFEST).in $(builddefs) @echo " [TOML] $@" $(Q)$(SED) -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@cargo_lto@|$(CARGO_LTO)|g" \ + -e "s|@cargo_cmt_gettext@|$(CARGO_CMT_GETTEXT)|g" \ < $< | sed -e 's|\~WIP.*|"|g' > $@ ifeq ($(USE_SYSTEM_CRATES),yes)
healer/src/main.rs +3 -1
@@ -71,7 +71,7 @@ impl App {/// Main app method fn main (&self) ->Result <ExitCode > { -let _fp =File ::open (&self.path).with_context (||"Opening filesystem failed" )?; +let _fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?;Ok (ExitCode ::SUCCESS ) }
@@ -89,6 +89,8 @@ impl From<Cli> for App { }fn main () ->ExitCode { + xfsprogs::init_localization (); +let args =Cli ::new ();if args.0 .get_flag ("version" ) {println! ("{} {}" ,M_ ("xfs_healer version" ), xfsprogs::VERSION );
healer/src/xfsprogs.rs.in +20 -0
@@ -3,10 +3,30 @@ * Copyright (C) 2025 Oracle. All Rights Reserved. * Author: Darrick J. Wong <[email protected]> */ +#![allow(unexpected_cfgs)] + pub const VERSION: &str = "@pkg_version@"; +/// Try to initialize a localization library. Like the other xfsprogs utilities, we don't care +/// if this fails. +#[cfg(feature = "gettext")] +pub fn init_localization() { + let _ = gettextrs::setlocale(gettextrs::LocaleCategory::LcAll, ""); + let _ = gettextrs::bindtextdomain("@PACKAGE@", "@LOCALEDIR@"); + let _ = gettextrs::textdomain("@PACKAGE@"); +} + +/// Look up a string in a message catalog +#[cfg(feature = "gettext")] +pub use gettextrs::gettext as M_; + +/// Pretend to initialize localization library +#[cfg(not(feature = "gettext"))] +pub fn init_localization() {} + /// Dummy function to simulate looking up a string in a message catalog #[allow(non_snake_case)] +#[cfg(not(feature = "gettext"))] pub fn M_<T: Into<String>>(msgid: T) -> String { msgid.into() }
include/builddefs.in +1 -0
@@ -102,6 +102,7 @@ ENABLE_GETTEXT = @enable_gettext@ ENABLE_EDITLINE = @enable_editline@ ENABLE_SCRUB = @enable_scrub@ ENABLE_HEALER = @enable_healer@ +ENABLE_GETTEXT_RS = @enable_gettext_rs@ HAVE_ZIPPED_MANPAGES = @have_zipped_manpages@
include/buildrules +1 -0
@@ -88,6 +88,7 @@ ifdef POTHEAD XGETTEXT_FLAGS=\ --keyword=_ \ --keyword=N_ \ + --keyword=M_ \ --package-name=$(PKG_NAME) \ --package-version=$(PKG_VERSION) \ --msgid-bugs-address=$(PKG_BUGREPORT)
m4/package_rust.m4 +6 -0
@@ -119,10 +119,16 @@ ENDL # Do we have all the crates we need for xfs_healer? AC_DEFUN([AC_HAVE_HEALER_CRATES], [ + if test "$enable_gettext_rs" = "yes"; then + gettext_dep='gettext-rs = { version = "0", optional = true }' # 0.7.0 + else + gettext_dep="" + fi AC_CHECK_CRATES([have_healer_crates], [xfs_healer], [ clap = { version = "4.0.32", features = [["derive"]] } anyhow = { version = "1.0.69" } +$gettext_dep ], [yes], [no]) ])
PATCH 3/19 xfs_healer: bindgen xfs_fs.h
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create Rust bindings for all the stuff in xfs_fs.h so that we can call the health monitor and online fsck ioctls later.
--- healer/bindgen_xfs_fs.h | 6 +++++ configure.ac | 14 +++++++++++- healer/Makefile | 14 ++++++++++++ healer/rbindgen | 57 +++++++++++++++++++++++++++++++++++++++++++++++ healer/src/lib.rs | 1 + include/builddefs.in | 3 ++ m4/package_rust.m4 | 22 ++++++++++++++++++ 7 files changed, 116 insertions(+), 1 deletion(-) create mode 100644 healer/bindgen_xfs_fs.h create mode 100755 healer/rbindgen
healer/bindgen_xfs_fs.h new file +6 -0
@@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (c) 2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +#include "xfs.h"
configure.ac +13 -1
@@ -283,7 +283,7 @@ if test "$have_rustc" = "yes"; then AC_HAVE_CARGO fi -# If we have cargo, check that our crate dependencies are present +# If we have cargo, check that we have the first two dependencies for bindgen if test "$have_cargo" = "yes"; then if test "$with_system_crates" = "yes"; then AC_USE_SYSTEM_CRATES
@@ -291,6 +291,18 @@ if test "$have_cargo" = "yes"; then AC_MAYBE_USE_SYSTEM_CRATES fi AC_HAVE_CLIPPY + AC_HAVE_CLANG + AC_HAVE_RUSTFMT +fi + +# If we have the first two deps for bindgen, check that we have bindgen +if test "$have_clang:$have_rustfmt" = "yes:yes"; then + AC_HAVE_BINDGEN +fi + +# If we have rustc, cargo, clang, rustfmt, and bindgen, check that our crate +# dependencies are present +if test "$have_bindgen" = "yes"; then AC_HAVE_HEALER_CRATES fi
healer/Makefile +14 -0
@@ -13,12 +13,17 @@ INSTALL_HEALER = install-healer # Rust implementation ifeq ($(HAVE_HEALER_CRATES),yes) +HFILES = \ + bindgen_xfs_fs.h + RUSTFILES = \ src/lib.rs \ src/main.rs \ + src/xfs_fs.rs \ src/xfsprogs.rs BUILT_RUSTFILES = \ + src/xfs_fs.rs \ src/xfsprogs.rs CARGO_MANIFEST=Cargo.toml
@@ -130,10 +135,19 @@ $(CARGO_CONFIG): touch $@ endif +ifeq ($(HAVE_RUSTFMT),yes) +rustfmt: $(RUSTFILES) + rustfmt $^ +endif + docs: @echo " [CARGO] doc $@" $(Q)cargo doc --no-deps +src/xfs_fs.rs: bindgen_xfs_fs.h ../libxfs/xfs_fs.h rbindgen + @echo " [RBIND] $@" + $(Q)./rbindgen $< $@ '*xfs_fs.h*' + # cargo install only knows how to build a binary and install it to $root/bin, # so we install it to ./rust/bin/ and let the install-rust target move it to # $prefix/usr/libexec/xfsprogs like we want.
healer/rbindgen new file +57 -0
57 lines not on this page · raw message
healer/src/lib.rs +1 -0
@@ -5,3 +5,4 @@ */pub mod xfsprogs; +pub mod xfs_fs;
include/builddefs.in +3 -0
@@ -25,6 +25,9 @@ HAVE_HEALER_CRATES = @have_healer_crates@ CARGOFLAGS = @CARGOFLAGS@ USE_SYSTEM_CRATES = @use_system_crates@ HAVE_CLIPPY = @have_clippy@ +HAVE_CLANG = @have_clang@ +HAVE_RUSTFMT = @have_rustfmt@ +HAVE_BINDGEN = @have_bindgen@ # make sure we don't pick up whacky LDFLAGS from the make environment and # only use what we calculate from the configured options above.
m4/package_rust.m4 +22 -0
@@ -132,3 +132,25 @@ $gettext_dep ], [yes], [no]) ]) + +# Check if clang is installed so that bindgen can find system headers. +AC_DEFUN([AC_HAVE_CLANG], +[ + AC_CHECK_PROG([have_clang], [clang], [yes], [no]) + AC_SUBST(have_clang) +]) + +# Check if rustfmt is installed; bindgen needs this to produce readable source +# code. +AC_DEFUN([AC_HAVE_RUSTFMT], +[ + AC_CHECK_PROG([have_rustfmt], [rustfmt], [yes], [no]) + AC_SUBST(have_rustfmt) +]) + +# Check if bindgen (aka the C FFI generator) is installed +AC_DEFUN([AC_HAVE_BINDGEN], +[ + AC_CHECK_PROG([have_bindgen], [bindgen], [yes], [no]) + AC_SUBST(have_bindgen) +])
PATCH 4/19 xfs_healer: define Rust objects for health events and kernel interface
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create our own Rust types for health events; these objects will know how to report themselves and (later) how to initiate repairs. Create the Rust binding to the kernel ioctl interface.
--- healer/Cargo.toml.in | 2 healer/Makefile | 9 +- healer/src/healthmon/event.rs | 82 ++++++++++++++ healer/src/healthmon/fs.rs | 146 ++++++++++++++++++++++++ healer/src/healthmon/groups.rs | 139 +++++++++++++++++++++++ healer/src/healthmon/inodes.rs | 119 ++++++++++++++++++++ healer/src/healthmon/mod.rs | 14 ++ healer/src/lib.rs | 3 + healer/src/main.rs | 5 + healer/src/util.rs | 72 ++++++++++++ healer/src/xfs_types.rs | 240 ++++++++++++++++++++++++++++++++++++++++ m4/package_rust.m4 | 2 12 files changed, 830 insertions(+), 3 deletions(-) create mode 100644 healer/src/healthmon/event.rs create mode 100644 healer/src/healthmon/fs.rs create mode 100644 healer/src/healthmon/groups.rs create mode 100644 healer/src/healthmon/inodes.rs create mode 100644 healer/src/healthmon/mod.rs create mode 100644 healer/src/util.rs create mode 100644 healer/src/xfs_types.rs
healer/Cargo.toml.in +2 -0
@@ -16,6 +16,7 @@ lto = @cargo_lto@ [dependencies] clap = { version = "4.0.32", features = ["derive"] } anyhow = { version = "1.0.69" } +enumset = { version = "1.0.12" } # XXX: Crates with major version 0 are not considered ABI-stable, so the minor # version is treated as if it were the major version. This creates problems
@@ -23,6 +24,7 @@ anyhow = { version = "1.0.69" } # version 0. Until these crates reach 1.0.0, we'll have to patch when things # break. Ref: # https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html +nix = { version = "0", features = ["ioctl"] } # 0.26.1 # Dynamically comment out all the gettextrs related dependency information in # Cargo.toml becuse cargo requires the crate to be present so that it can
healer/Makefile +8 -1
@@ -19,8 +19,15 @@ HFILES = \ RUSTFILES = \ src/lib.rs \ src/main.rs \ + src/util.rs \ src/xfs_fs.rs \ - src/xfsprogs.rs + src/xfsprogs.rs \ + src/xfs_types.rs \ + src/healthmon/event.rs \ + src/healthmon/fs.rs \ + src/healthmon/groups.rs \ + src/healthmon/inodes.rs \ + src/healthmon/mod.rs BUILT_RUSTFILES = \ src/xfs_fs.rs \
healer/src/healthmon/event.rs new file +82 -0
82 lines not on this page · raw message
healer/src/healthmon/fs.rs new file +146 -0
146 lines not on this page · raw message
healer/src/healthmon/groups.rs new file +139 -0
139 lines not on this page · raw message
healer/src/healthmon/inodes.rs new file +119 -0
119 lines not on this page · raw message
healer/src/healthmon/mod.rs new file +14 -0
@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright (C) 2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +use crate ::xfs_fs::xfs_health_monitor; +use nix::ioctl_write_ptr; + +pub mod event; +pub mod fs; +pub mod groups; +pub mod inodes; + +ioctl_write_ptr! (xfs_ioc_health_monitor,'X' ,68 , xfs_health_monitor);
healer/src/lib.rs +3 -0
@@ -6,3 +6,6 @@pub mod xfsprogs;pub mod xfs_fs; +pub mod xfs_types; +pub mod util; +pub mod healthmon;
healer/src/main.rs +3 -2
@@ -8,6 +8,7 @@ use clap::{value_parser, Arg, ArgAction, ArgMatches, Command};use std::fs::File ;use std::path::PathBuf ;use std::process::ExitCode ; +use xfs_healer::printlogln;use xfs_healer::xfsprogs;use xfs_healer::xfsprogs::M_ ;
@@ -93,12 +94,12 @@ fn main() -> ExitCode {let args =Cli ::new ();if args.0 .get_flag ("version" ) { -println! ("{} {}" ,M_ ("xfs_healer version" ), xfsprogs::VERSION ); +printlogln! ("{} {}" ,M_ ("xfs_healer version" ), xfsprogs::VERSION );return ExitCode ::SUCCESS ; }if args.0 .get_flag ("debug" ) { -println! ("args: {:?}" , args); +printlogln! ("args: {:?}" , args); }let app:App = args.into ();
healer/src/util.rs new file +72 -0
72 lines not on this page · raw message
healer/src/xfs_types.rs new file +240 -0
240 lines not on this page · raw message
m4/package_rust.m4 +2 -0
@@ -129,6 +129,8 @@ AC_DEFUN([AC_HAVE_HEALER_CRATES], clap = { version = "4.0.32", features = [["derive"]] } anyhow = { version = "1.0.69" } $gettext_dep +nix = { version = "0", features = [["ioctl"]] } # 0.26.1 +enumset = { version = "1.0.12" } ], [yes], [no]) ])
PATCH 5/19 xfs_healer: read binary health events from the kernel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Decode binary health events objects read from the kernel into the corresponding Rust objects so that we can deal with the events.
--- healer/Makefile | 1 healer/src/healthmon/cstruct.rs | 343 +++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 1 healer/src/main.rs | 25 +++ 4 files changed, 369 insertions(+), 1 deletion(-) create mode 100644 healer/src/healthmon/cstruct.rs
healer/Makefile +1 -0
@@ -23,6 +23,7 @@ RUSTFILES = \ src/xfs_fs.rs \ src/xfsprogs.rs \ src/xfs_types.rs \ + src/healthmon/cstruct.rs \ src/healthmon/event.rs \ src/healthmon/fs.rs \ src/healthmon/groups.rs \
healer/src/healthmon/cstruct.rs new file +343 -0
343 lines not on this page · raw message
healer/src/healthmon/mod.rs +1 -0
@@ -6,6 +6,7 @@use crate ::xfs_fs::xfs_health_monitor;use nix::ioctl_write_ptr; +pub mod cstruct;pub mod event;pub mod fs;pub mod groups;
healer/src/main.rs +24 -1
@@ -8,6 +8,8 @@ use clap::{value_parser, Arg, ArgAction, ArgMatches, Command};use std::fs::File ;use std::path::PathBuf ;use std::process::ExitCode ; +use xfs_healer::healthmon::cstruct::CStructMonitor ; +use xfs_healer::healthmon::event::XfsHealthEvent ;use xfs_healer::printlogln;use xfs_healer::xfsprogs;use xfs_healer::xfsprogs::M_ ;
@@ -70,9 +72,30 @@ impl App { self.path.display ().to_string () } +/// Handle a health event that has been decoded into real objects +fn process_event (&self, cooked:Result <Box <dyn XfsHealthEvent >>) { +match cooked { +Err (e) => { +eprintln! ("{}: {:#}" , self.path.display (), e) + } +Ok (event) => { +if self.log || event.must_log () { +printlogln! ("{}: {}" , self.path.display (), event.format ()); + } + } + } + } +/// Main app method fn main (&self) ->Result <ExitCode > { -let _fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; +let fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; + +let hmon =CStructMonitor ::try_new (fp, &self.path, self.everything) + .with_context (||M_ ("Opening health monitor file" ))?; + +for raw_eventin hmon { + self.process_event (raw_event.cook ()); + }Ok (ExitCode ::SUCCESS ) }
PATCH 6/19 xfs_healer: read json health events from the kernel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The kernel can give us filesystem health events in json, so let's use the json deserializer to turn them into Rust associative arrays and return them from our iterator. This isn't totally necessary since we have the C structure variant, but it'll help us test the other interface. Note that we use a fair amount of EnumString magic to automatically provide translators for the json.
--- healer/Cargo.toml.in | 3 healer/Makefile | 1 healer/src/healthmon/event.rs | 4 healer/src/healthmon/fs.rs | 7 + healer/src/healthmon/groups.rs | 7 + healer/src/healthmon/inodes.rs | 7 + healer/src/healthmon/json.rs | 398 ++++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 1 healer/src/main.rs | 26 ++- healer/src/xfs_types.rs | 8 + m4/package_rust.m4 | 3 11 files changed, 453 insertions(+), 12 deletions(-) create mode 100644 healer/src/healthmon/json.rs
healer/Cargo.toml.in +3 -0
@@ -17,6 +17,7 @@ lto = @cargo_lto@ clap = { version = "4.0.32", features = ["derive"] } anyhow = { version = "1.0.69" } enumset = { version = "1.0.12" } +serde_json = { version = "1.0.87" } # XXX: Crates with major version 0 are not considered ABI-stable, so the minor # version is treated as if it were the major version. This creates problems
@@ -25,6 +26,8 @@ enumset = { version = "1.0.12" } # break. Ref: # https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html nix = { version = "0", features = ["ioctl"] } # 0.26.1 +strum = { version = "0" } # 0.19.2 +strum_macros = { version = "0" } # 0.19.2 # Dynamically comment out all the gettextrs related dependency information in # Cargo.toml becuse cargo requires the crate to be present so that it can
healer/Makefile +1 -0
@@ -28,6 +28,7 @@ RUSTFILES = \ src/healthmon/fs.rs \ src/healthmon/groups.rs \ src/healthmon/inodes.rs \ + src/healthmon/json.rs \ src/healthmon/mod.rs BUILT_RUSTFILES = \
healer/src/healthmon/event.rs +3 -1
@@ -5,6 +5,7 @@ */use crate ::display_for_enum;use crate ::xfsprogs::M_ ; +use strum_macros::EnumString ;/// Common behaviors of all health events pub trait XfsHealthEvent {
@@ -18,7 +19,8 @@ pub trait XfsHealthEvent { }/// Health status for metadata events -#[derive( Debug )] +#[derive( Debug , EnumString )] +#[strum(serialize_all = "lowercase" )] pub enum XfsHealthStatus {/// Problems have been observed at runtime Sick ,
healer/src/healthmon/fs.rs +5 -2
@@ -11,9 +11,11 @@ use crate::xfs_types::XfsPhysRange;use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use strum_macros::EnumString ;/// Metadata types for an XFS whole-fs metadata -#[derive( EnumSetType , Debug )] +#[derive( EnumSetType , Debug , EnumString )] +#[strum(serialize_all = "lowercase" )] pub enum XfsWholeFsMetadata {FsCounters ,GrpQuota ,
@@ -65,7 +67,8 @@ impl XfsHealthEvent for XfsWholeFsEvent { }/// Reasons for a filesystem shutdown event -#[derive( EnumSetType , Debug )] +#[derive( EnumSetType , Debug , EnumString )] +#[strum(serialize_all = "snake_case" )] pub enum XfsShutdownReason {CorruptIncore ,CorruptOndisk ,
healer/src/healthmon/groups.rs +5 -2
@@ -11,9 +11,11 @@ use crate::xfs_types::{XfsAgNumber, XfsRgNumber};use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use strum_macros::EnumString ;/// Metadata types for an allocation group on the data device -#[derive( EnumSetType , Debug )] +#[derive( EnumSetType , Debug , EnumString )] +#[strum(serialize_all = "lowercase" )] pub enum XfsPeragMetadata {Agf ,Agfl ,
@@ -82,7 +84,8 @@ impl XfsHealthEvent for XfsPeragEvent { }/// Metadata types for an allocation group on the realtime device -#[derive( EnumSetType , Debug )] +#[derive( EnumSetType , Debug , EnumString )] +#[strum(serialize_all = "lowercase" )] pub enum XfsRtgroupMetadata {Bitmap ,Summary ,
healer/src/healthmon/inodes.rs +5 -2
@@ -11,9 +11,11 @@ use crate::xfs_types::{XfsFid, XfsFileRange};use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use strum_macros::EnumString ;/// Metadata types for an XFS inode -#[derive( EnumSetType , Debug )] +#[derive( EnumSetType , Debug , EnumString )] +#[strum(serialize_all = "lowercase" )] pub enum XfsInodeMetadata {Bmapbta ,Bmapbtc ,
@@ -73,7 +75,8 @@ impl XfsHealthEvent for XfsInodeEvent { }/// File I/O types -#[derive( Debug )] +#[derive( Debug , EnumString )] +#[strum(serialize_all = "snake_case" )] pub enum XfsFileIoErrorType {Readahead ,Writeback ,
healer/src/healthmon/json.rs new file +398 -0
398 lines not on this page · raw message
healer/src/healthmon/mod.rs +1 -0
@@ -11,5 +11,6 @@ pub mod event;pub mod fs;pub mod groups;pub mod inodes; +pub mod json;ioctl_write_ptr! (xfs_ioc_health_monitor,'X' ,68 , xfs_health_monitor);
healer/src/main.rs +22 -4
@@ -10,6 +10,7 @@ use std::path::PathBuf;use std::process::ExitCode ;use xfs_healer::healthmon::cstruct::CStructMonitor ;use xfs_healer::healthmon::event::XfsHealthEvent ; +use xfs_healer::healthmon::json::JsonMonitor ;use xfs_healer::printlogln;use xfs_healer::xfsprogs;use xfs_healer::xfsprogs::M_ ;
@@ -53,6 +54,12 @@ impl Cli { .value_parser(value_parser!(PathBuf)) .required_unless_present("version"), ) + .arg ( +Arg ::new ("json" ) + .long ("json" ) + .help (M_ ("Use the JSON kernel interface instead of C" )) + .action (ArgAction ::SetTrue ), + ) .get_matches()) } }
@@ -63,6 +70,7 @@ struct App { debug: bool, log: bool, everything: bool, + json: bool, path: PathBuf, }
@@ -90,11 +98,20 @@ impl App {fn main (&self) ->Result <ExitCode > {let fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; -let hmon =CStructMonitor ::try_new (fp, &self.path, self.everything) - .with_context (||M_ ("Opening health monitor file" ))?; +if self.json { +let hmon =JsonMonitor ::try_new (fp, &self.path, self.everything, self.debug) + .with_context (||M_ ("Opening js health monitor file" ))?; -for raw_eventin hmon { - self.process_event (raw_event.cook ()); +for raw_eventin hmon { + self.process_event (raw_event.cook ()); + } + }else { +let hmon =CStructMonitor ::try_new (fp, &self.path, self.everything) + .with_context (||M_ ("Opening health monitor file" ))?; + +for raw_eventin hmon { + self.process_event (raw_event.cook ()); + } }Ok (ExitCode ::SUCCESS )
@@ -108,6 +125,7 @@ impl From<Cli> for App { log: cli.0.get_flag("log"), everything: cli.0.get_flag("everything"), path: cli.0.get_one::<PathBuf>("path").unwrap().to_path_buf(), + json: cli.0.get_flag("json"), } } }
healer/src/xfs_types.rs +7 -1
@@ -9,6 +9,7 @@ use crate::xfsprogs::M_;use anyhow::{Error ,Result };use std::fmt::Display ;use std::fmt::Formatter ; +use strum_macros::EnumString ;/// Allocation group number on the data device #[derive( Debug )]
@@ -55,10 +56,15 @@ impl TryFrom<u64> for XfsRgNumber { }/// Disk devices -#[derive( Debug )] +#[derive( Debug , EnumString )] pub enum XfsDevice { +#[strum(serialize = "datadev" )] Data , + +#[strum(serialize = "logdev" )] Log , + +#[strum(serialize = "rtdev" )] Realtime , }
m4/package_rust.m4 +3 -0
@@ -131,6 +131,9 @@ anyhow = { version = "1.0.69" } $gettext_dep nix = { version = "0", features = [["ioctl"]] } # 0.26.1 enumset = { version = "1.0.12" } +strum = { version = "0" } # 0.19.2 +strum_macros = { version = "0" } # 0.19.2 +serde_json = { version = "1.0.87" } ], [yes], [no]) ])
PATCH 7/19 xfs_healer: create a weak file handle so we don't pin the mount
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a custom file handle object that allows us to maintain a "soft" reference to a mounted filesystem. The purpose of this is to avoid pinning the mount while xfs_healer runs by not leaving an open fd while retaining the ability to reconnect to the filesystem later so that we can look up paths for reporting, and run repairs.
This means that the filesystem must still be available at the same path at reconnect time, which may result in the program exiting if mount --move is used.
Note that we open-code the XFS_IOC_FD_TO_HANDLE call to avoid overcomplicating the cargo configuration to link with ../libhandle/libhandle.la as a static import for a single function call.
--- healer/Makefile | 1 healer/src/lib.rs | 1 healer/src/main.rs | 3 + healer/src/weakhandle.rs | 101 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 106 insertions(+) create mode 100644 healer/src/weakhandle.rs
healer/Makefile +1 -0
@@ -20,6 +20,7 @@ RUSTFILES = \ src/lib.rs \ src/main.rs \ src/util.rs \ + src/weakhandle.rs \ src/xfs_fs.rs \ src/xfsprogs.rs \ src/xfs_types.rs \
healer/src/lib.rs +1 -0
@@ -9,3 +9,4 @@ pub mod xfs_fs;pub mod xfs_types;pub mod util;pub mod healthmon; +pub mod weakhandle;
healer/src/main.rs +3 -0
@@ -12,6 +12,7 @@ use xfs_healer::healthmon::cstruct::CStructMonitor;use xfs_healer::healthmon::event::XfsHealthEvent ;use xfs_healer::healthmon::json::JsonMonitor ;use xfs_healer::printlogln; +use xfs_healer::weakhandle::WeakHandle ;use xfs_healer::xfsprogs;use xfs_healer::xfsprogs::M_ ;
@@ -97,6 +98,8 @@ impl App {/// Main app method fn main (&self) ->Result <ExitCode > {let fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; +let _fh =WeakHandle ::try_new (&fp, &self.path) + .with_context (||M_ ("Configuring filesystem handle" ))?;if self.json {let hmon =JsonMonitor ::try_new (fp, &self.path, self.everything, self.debug)
healer/src/weakhandle.rs new file +101 -0
101 lines not on this page · raw message
PATCH 8/19 xfs_healer: fix broken filesystem metadata
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use the soft file handle we created in the previous patch to schedule repairs when possible.
--- healer/Makefile | 1 healer/src/healthmon/event.rs | 29 ++++ healer/src/healthmon/fs.rs | 6 + healer/src/healthmon/groups.rs | 10 + healer/src/healthmon/inodes.rs | 6 + healer/src/lib.rs | 1 healer/src/main.rs | 21 ++- healer/src/repair.rs | 302 ++++++++++++++++++++++++++++++++++++++++ healer/src/weakhandle.rs | 13 ++ healer/src/xfs_types.rs | 56 +++++++ 10 files changed, 435 insertions(+), 10 deletions(-) create mode 100644 healer/src/repair.rs
healer/Makefile +1 -0
@@ -19,6 +19,7 @@ HFILES = \ RUSTFILES = \ src/lib.rs \ src/main.rs \ + src/repair.rs \ src/util.rs \ src/weakhandle.rs \ src/xfs_fs.rs \
healer/src/healthmon/event.rs +28 -1
@@ -4,6 +4,7 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::display_for_enum; +use crate ::repair::Repair ;use crate ::xfsprogs::M_ ;use strum_macros::EnumString ;
@@ -16,10 +17,36 @@ pub trait XfsHealthEvent {/// Format this event as something we can display fn format (&self) ->String ; + +/// Generate the inputs to a kernel scrub ioctl +fn schedule_repairs (&self) ->Vec <Repair > { +vec! [] + } } +/// Boilerplate implementation of a schedule_repairs function. Pass a lambda +/// that generates a Repair object from &self and sm_type. +#[macro_export] +macro_rules! schedule_repairs { + ($event_type:ty , $lambda: expr ) => { +fn schedule_repairs (&self) ->Vec <$crate::repair::Repair > { +if self.status != $crate::healthmon::event::XfsHealthStatus ::Sick { +return vec! []; + } +let mut ret =Vec ::new (); +for fin self.metadata { +if let Some (sm_type) = f.to_scrub () { + ret.push ($lambda(self, sm_type)); + } + } + ret + } + }; +} +pub (crate )use schedule_repairs; +/// Health status for metadata events -#[derive( Debug , EnumString )] +#[derive( PartialEq , Debug , EnumString )] #[strum(serialize_all = "lowercase" )] pub enum XfsHealthStatus {/// Problems have been observed at runtime
healer/src/healthmon/fs.rs +6 -0
@@ -4,8 +4,10 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::display_for_enum; +use crate ::healthmon::event::schedule_repairs;use crate ::healthmon::event::XfsHealthEvent ;use crate ::healthmon::event::XfsHealthStatus ; +use crate ::repair::Repair ;use crate ::util::format_set;use crate ::xfs_types::XfsPhysRange ;use crate ::xfsprogs::M_ ;
@@ -64,6 +66,10 @@ impl XfsHealthEvent for XfsWholeFsEvent { self.status ) } + +schedule_repairs! (XfsWholeFsEvent , |_: &XfsWholeFsEvent , sm_type| { +Repair ::from_whole_fs (sm_type) + }); }/// Reasons for a filesystem shutdown event
healer/src/healthmon/groups.rs +10 -0
@@ -4,8 +4,10 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::display_for_enum; +use crate ::healthmon::event::schedule_repairs;use crate ::healthmon::event::XfsHealthEvent ;use crate ::healthmon::event::XfsHealthStatus ; +use crate ::repair::Repair ;use crate ::util::format_set;use crate ::xfs_types::{XfsAgNumber ,XfsRgNumber };use crate ::xfsprogs::M_ ;
@@ -81,6 +83,10 @@ impl XfsHealthEvent for XfsPeragEvent { self.status ) } + +schedule_repairs! (XfsPeragEvent , |s: &XfsPeragEvent , sm_type| { +Repair ::from_perag (sm_type, s.group) + }); }/// Metadata types for an allocation group on the realtime device
@@ -139,4 +145,8 @@ impl XfsHealthEvent for XfsRtgroupEvent { self.status ) } + +schedule_repairs! (XfsRtgroupEvent , |s: &XfsRtgroupEvent , sm_type| { +Repair ::from_rtgroup (sm_type, s.group) + }); }
healer/src/healthmon/inodes.rs +6 -0
@@ -4,8 +4,10 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::display_for_enum; +use crate ::healthmon::event::schedule_repairs;use crate ::healthmon::event::XfsHealthEvent ;use crate ::healthmon::event::XfsHealthStatus ; +use crate ::repair::Repair ;use crate ::util::format_set;use crate ::xfs_types::{XfsFid ,XfsFileRange };use crate ::xfsprogs::M_ ;
@@ -72,6 +74,10 @@ impl XfsHealthEvent for XfsInodeEvent {fn format (&self) ->String {format! ("{} {} {}" , self.fid,format_set (self.metadata), self.status) } + +schedule_repairs! (XfsInodeEvent , |s: &XfsInodeEvent , sm_type| { +Repair ::from_file (sm_type, s.fid) + }); }/// File I/O types
healer/src/lib.rs +1 -0
@@ -10,3 +10,4 @@ pub mod xfs_types;pub mod util;pub mod healthmon;pub mod weakhandle; +pub mod repair;
healer/src/main.rs +17 -4
@@ -61,6 +61,12 @@ impl Cli { .help(M_("Use the JSON kernel interface instead of C")) .action(ArgAction::SetTrue), ) + .arg ( +Arg ::new ("repair" ) + .long ("repair" ) + .help (M_ ("Always repair corrupt metadata" )) + .action (ArgAction ::SetTrue ), + ) .get_matches()) } }
@@ -72,6 +78,7 @@ struct App { log: bool, everything: bool, json: bool, + repair: bool, path: PathBuf, }
@@ -82,7 +89,7 @@ impl App { }/// Handle a health event that has been decoded into real objects -fn process_event (&self, cooked:Result <Box <dyn XfsHealthEvent >>) { +fn process_event (&self, fh: &WeakHandle , cooked:Result <Box <dyn XfsHealthEvent >>) {match cooked {Err (e) => {eprintln! ("{}: {:#}" , self.path.display (), e)
@@ -91,6 +98,11 @@ impl App {if self.log || event.must_log () {printlogln! ("{}: {}" , self.path.display (), event.format ()); } +if self.repair { +for mut repairin event.schedule_repairs () { + repair.perform (fh) + } + } } } }
@@ -98,7 +110,7 @@ impl App {/// Main app method fn main (&self) ->Result <ExitCode > {let fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; -let _fh =WeakHandle ::try_new (&fp, &self.path) +let fh =WeakHandle ::try_new (&fp, &self.path) .with_context (||M_ ("Configuring filesystem handle" ))?;if self.json {
@@ -106,14 +118,14 @@ impl App { .with_context(|| M_("Opening js health monitor file"))?;for raw_eventin hmon { - self.process_event (raw_event.cook ()); + self.process_event (&fh, raw_event.cook ()); } }else {let hmon =CStructMonitor ::try_new (fp, &self.path, self.everything) .with_context (||M_ ("Opening health monitor file" ))?;for raw_eventin hmon { - self.process_event (raw_event.cook ()); + self.process_event (&fh, raw_event.cook ()); } }
@@ -129,6 +141,7 @@ impl From<Cli> for App { everything: cli.0.get_flag("everything"), path: cli.0.get_one::<PathBuf>("path").unwrap().to_path_buf(), json: cli.0.get_flag("json"), + repair: cli.0.get_flag("repair"), } } }
healer/src/repair.rs new file +302 -0
302 lines not on this page · raw message
healer/src/weakhandle.rs +13 -0
@@ -10,6 +10,8 @@ use crate::xfsprogs::M_;use anyhow::{Error ,Result };use nix::ioctl_readwrite;use nix::libc::O_LARGEFILE ; +use std::fmt::Display ; +use std::fmt::Formatter ;use std::fs::File ;use std::io::ErrorKind ;use std::os::fd::AsRawFd ;
@@ -91,6 +93,11 @@ impl WeakHandle<'_> {Ok (fp) } +/// Report mountpoint in a displayable manner +pub fn mountpoint (&self) ->String { + self.mountpoint.display ().to_string () + } +/// Create a soft handle from an open file descriptor and its mount point pub fn try_new <'a>(fp: &File , mountpoint: &'aPath ) ->Result <WeakHandle <'a>> {Ok (WeakHandle {
@@ -99,3 +106,9 @@ impl WeakHandle<'_> { }) } } + +impl Display for WeakHandle <'_> { +fn fmt (&self, f: &mut Formatter ) -> std::fmt::Result { +write! (f,"{}" , self.mountpoint.display ()) + } +}
healer/src/xfs_types.rs +51 -5
107 lines not on this page · raw message
PATCH 9/19 xfs_healer: check for fs features needed for effective repairs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Online repair relies heavily on back references such as reverse mappings and directory parent pointers to add redundancy to the filesystem. Make the rust program check for these two features and whine a bit if they are missing.
--- healer/Makefile | 1 + healer/src/fsgeom.rs | 41 +++++++++++++++++++++++++++++++++++++++++ healer/src/lib.rs | 1 + healer/src/main.rs | 41 +++++++++++++++++++++++++++++++++++++++++ healer/src/repair.rs | 13 +++++++++++++ 5 files changed, 97 insertions(+) create mode 100644 healer/src/fsgeom.rs
healer/Makefile +1 -0
@@ -17,6 +17,7 @@ HFILES = \ bindgen_xfs_fs.h RUSTFILES = \ + src/fsgeom.rs \ src/lib.rs \ src/main.rs \ src/repair.rs \
healer/src/fsgeom.rs new file +41 -0
41 lines not on this page · raw message
healer/src/lib.rs +1 -0
@@ -11,3 +11,4 @@ pub mod util;pub mod healthmon;pub mod weakhandle;pub mod repair; +pub mod fsgeom;
healer/src/main.rs +41 -0
57 lines not on this page · raw message
healer/src/repair.rs +13 -0
@@ -15,6 +15,7 @@ use crate::xfs_types::{XfsAgNumber, XfsFid, XfsRgNumber};use crate ::xfsprogs::M_ ;use anyhow::{Context ,Result };use nix::ioctl_readwrite; +use std::fs::File ;use std::os::fd::AsRawFd ;ioctl_readwrite! (xfs_ioc_scrub_metadata,'X' ,60 , xfs_scrub_metadata);
@@ -172,6 +173,18 @@ pub struct Repair { }impl Repair { +/// Determine if repairs are supported by this kernel +pub fn is_supported (fp: &File ) ->bool { +let mut detail =xfs_scrub_metadata { + sm_type: xfs_fs::XFS_SCRUB_TYPE_PROBE , + sm_flags: xfs_fs::XFS_SCRUB_IFLAG_REPAIR , + ..Default ::default () + }; + +// SAFETY: Trusting the kernel not to corrupt memory. +unsafe {xfs_ioc_scrub_metadata (fp.as_raw_fd (), &mut detail).is_ok () } + } +/// Schedule a full-filesystem metadata repair pub fn from_whole_fs (t:XfsScrubType ) ->Repair {Repair {
PATCH 10/19 xfs_healer: use getparents to look up file names
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If the kernel tells about something that happened to a file, use the GETPARENTS ioctl to try to look up the path to that file for more ergonomic reporting. In Rust.
--- healer/Makefile | 1 healer/src/getparents.rs | 210 ++++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/event.rs | 19 ++-- healer/src/healthmon/fs.rs | 38 ++++--- healer/src/healthmon/groups.rs | 32 ++++-- healer/src/healthmon/inodes.rs | 22 +++- healer/src/lib.rs | 1 healer/src/main.rs | 8 +- healer/src/repair.rs | 22 ++++ healer/src/weakhandle.rs | 30 ++++++ 10 files changed, 339 insertions(+), 44 deletions(-) create mode 100644 healer/src/getparents.rs
healer/Makefile +1 -0
@@ -18,6 +18,7 @@ HFILES = \ RUSTFILES = \ src/fsgeom.rs \ + src/getparents.rs \ src/lib.rs \ src/main.rs \ src/repair.rs \
healer/src/getparents.rs new file +210 -0
210 lines not on this page · raw message
healer/src/healthmon/event.rs +11 -8
@@ -5,7 +5,9 @@ */use crate ::display_for_enum;use crate ::repair::Repair ; +use crate ::weakhandle::WeakHandle ;use crate ::xfsprogs::M_ ; +use std::path::PathBuf ;use strum_macros::EnumString ;/// Common behaviors of all health events
@@ -15,8 +17,9 @@ pub trait XfsHealthEvent {false } -/// Format this event as something we can display -fn format (&self) ->String ; +/// Format this event as something we can display. Returns an optional +/// pathname string, and the message. +fn format (&self, fh: &WeakHandle ) -> (Option <PathBuf >,String );/// Generate the inputs to a kernel scrub ioctl fn schedule_repairs (&self) ->Vec <Repair > {
@@ -83,8 +86,8 @@ impl XfsHealthEvent for LostEvent {true } -fn format (&self) ->String { -format! ("{} {}" , self.count,M_ ("events lost" )) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + (None ,format! ("{} {}" , self.count,M_ ("events lost" ))) } }
@@ -92,8 +95,8 @@ impl XfsHealthEvent for LostEvent {pub struct RunningEvent {}impl XfsHealthEvent for RunningEvent { -fn format (&self) ->String { -M_ ("monitoring started" ) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + (None ,M_ ("monitoring started" )) } }
@@ -105,7 +108,7 @@ impl XfsHealthEvent for UnknownEvent {true } -fn format (&self) ->String { -M_ ("unrecognized event" ) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + (None ,M_ ("unrecognized event" )) } }
healer/src/healthmon/fs.rs +23 -15
@@ -9,10 +9,12 @@ use crate::healthmon::event::XfsHealthEvent;use crate ::healthmon::event::XfsHealthStatus ;use crate ::repair::Repair ;use crate ::util::format_set; +use crate ::weakhandle::WeakHandle ;use crate ::xfs_types::XfsPhysRange ;use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use std::path::PathBuf ;use strum_macros::EnumString ;/// Metadata types for an XFS whole-fs metadata
@@ -58,12 +60,15 @@ impl XfsWholeFsEvent { }impl XfsHealthEvent for XfsWholeFsEvent { -fn format (&self) ->String { -format! ( -"{} {} {}" , -format_set (self.metadata), -M_ ("status" ), - self.status +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + ( +None , +format! ( +"{} {} {}" , +format_set (self.metadata), +M_ ("status" ), + self.status + ), ) }
@@ -112,11 +117,14 @@ impl XfsHealthEvent for XfsShutdownEvent {true } -fn format (&self) ->String { -format! ( -"{} {}" , -M_ ("filesystem shut down due to" ), -format_set (self.reasons) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + ( +None , +format! ( +"{} {}" , +M_ ("filesystem shut down due to" ), +format_set (self.reasons) + ), ) } }
@@ -129,8 +137,8 @@ impl XfsHealthEvent for XfsUnmountEvent {true } -fn format (&self) ->String { -M_ ("filesystem unmounted" ) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + (None ,M_ ("filesystem unmounted" )) } }
@@ -149,7 +157,7 @@ impl XfsMediaErrorEvent { }impl XfsHealthEvent for XfsMediaErrorEvent { -fn format (&self) ->String { -format! ("{} {}" ,M_ ("media error on" ), self.range) +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + (None ,format! ("{} {}" ,M_ ("media error on" ), self.range)) } }
healer/src/healthmon/groups.rs +20 -12
@@ -9,10 +9,12 @@ use crate::healthmon::event::XfsHealthEvent;use crate ::healthmon::event::XfsHealthStatus ;use crate ::repair::Repair ;use crate ::util::format_set; +use crate ::weakhandle::WeakHandle ;use crate ::xfs_types::{XfsAgNumber ,XfsRgNumber };use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use std::path::PathBuf ;use strum_macros::EnumString ;/// Metadata types for an allocation group on the data device
@@ -75,12 +77,15 @@ impl XfsPeragEvent { }impl XfsHealthEvent for XfsPeragEvent { -fn format (&self) ->String { -format! ( -"{} {} {}" , - self.group, -format_set (self.metadata), - self.status +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + ( +None , +format! ( +"{} {} {}" , + self.group, +format_set (self.metadata), + self.status + ), ) }
@@ -137,12 +142,15 @@ impl XfsRtgroupEvent { }impl XfsHealthEvent for XfsRtgroupEvent { -fn format (&self) ->String { -format! ( -"{} {} {}" , - self.group, -format_set (self.metadata), - self.status +fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { + ( +None , +format! ( +"{} {} {}" , + self.group, +format_set (self.metadata), + self.status + ), ) }
healer/src/healthmon/inodes.rs +18 -4
@@ -9,10 +9,12 @@ use crate::healthmon::event::XfsHealthEvent;use crate ::healthmon::event::XfsHealthStatus ;use crate ::repair::Repair ;use crate ::util::format_set; +use crate ::weakhandle::WeakHandle ;use crate ::xfs_types::{XfsFid ,XfsFileRange };use crate ::xfsprogs::M_ ;use enumset::EnumSet ;use enumset::EnumSetType ; +use std::path::PathBuf ;use strum_macros::EnumString ;/// Metadata types for an XFS inode
@@ -71,8 +73,17 @@ impl XfsInodeEvent { }impl XfsHealthEvent for XfsInodeEvent { -fn format (&self) ->String { -format! ("{} {} {}" , self.fid,format_set (self.metadata), self.status) +fn format (&self, fh: &WeakHandle ) -> (Option <PathBuf >,String ) { +match fh.path_for (self.fid) { +Some (path) => ( +Some (path), +format! ("{} {}" ,format_set (self.metadata), self.status), + ), +None => ( +None , +format! ("{} {} {}" , self.fid,format_set (self.metadata), self.status), + ), + } }schedule_repairs! (XfsInodeEvent , |s: &XfsInodeEvent , sm_type| {
@@ -122,7 +133,10 @@ impl XfsFileIoErrorEvent { }impl XfsHealthEvent for XfsFileIoErrorEvent { -fn format (&self) ->String { -format! ("{} {} {}" , self.fid, self.iotype, self.range) +fn format (&self, fh: &WeakHandle ) -> (Option <PathBuf >,String ) { +match fh.path_for (self.fid) { +Some (path) => (Some (path),format! ("{} {}" , self.iotype, self.range)), +None => (None ,format! ("{} {} {}" , self.fid, self.iotype, self.range)), + } } }
healer/src/lib.rs +1 -0
@@ -12,3 +12,4 @@ pub mod healthmon;pub mod weakhandle;pub mod repair;pub mod fsgeom; +pub mod getparents;
healer/src/main.rs +6 -2
@@ -98,7 +98,11 @@ impl App { } Ok(event) => {if self.log || event.must_log () { -printlogln! ("{}: {}" , self.path.display (), event.format ()); +let (maybe_path, message) = event.format (fh); +match maybe_path { +Some (x) =>printlogln! ("{}: {}" , x.display (), message), +None =>printlogln! ("{}: {}" , self.path.display (), message), + }; }if self.repair {for mut repairin event.schedule_repairs () {
@@ -151,7 +155,7 @@ impl App { } } -let fh =WeakHandle ::try_new (&fp, &self.path) +let fh =WeakHandle ::try_new (&fp, &self.path, fsgeom) .with_context (||M_ ("Configuring filesystem handle" ))?;if self.json {
healer/src/repair.rs +20 -2
@@ -285,6 +285,19 @@ impl Repair { } } +/// Translate the target of this repair into a filesystem path +fn repair_path (&self, fh: &WeakHandle ) ->String { +if let RepairGroup ::File = self.group { +let fid:XfsFid = self.detail.into (); + +if let Some (path) = fh.path_for (fid) { +return path.display ().to_string (); + } + } + + fh.mountpoint () + } +/// Call the kernel to repair things fn repair (&mut self, fh: &WeakHandle ) ->Result <bool > {let fp = fh
@@ -305,10 +318,15 @@ impl Repair {pub fn perform (&mut self, fh: &WeakHandle ) {match self.repair (fh) {Err (e) => { -eprintln! ("{}: {:#}" , fh.mountpoint (), e); +eprintln! ("{}: {:#}" , self.repair_path (fh), e); } _ => { -printlogln! ("{}: {}: {}" , fh.mountpoint (), self.summary (), self.outcome); +printlogln! ( +"{}: {}: {}" , + self.repair_path (fh), + self.summary (), + self.outcome + ); } }; }
healer/src/weakhandle.rs +29 -1
@@ -4,8 +4,11 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::baddata; +use crate ::xfs_fs::xfs_fid; +use crate ::xfs_fs::xfs_fsop_geom;use crate ::xfs_fs::xfs_fsop_handlereq;use crate ::xfs_fs::xfs_handle; +use crate ::xfs_types::XfsFid ;use crate ::xfsprogs::M_ ;use anyhow::{Error ,Result };use nix::ioctl_readwrite;
@@ -73,6 +76,9 @@ pub struct WeakHandle<'a> {/// Filesystem handle handle: xfs_handle, + +/// Does this filesystem support parent pointers? + has_parent: bool, }impl WeakHandle <'_> {
@@ -99,12 +105,34 @@ impl WeakHandle<'_> { }/// Create a soft handle from an open file descriptor and its mount point -pub fn try_new <'a>(fp: &File , mountpoint: &'aPath ) ->Result <WeakHandle <'a>> { +pub fn try_new <'a>( + fp: &File , + mountpoint: &'aPath , + fsgeom:xfs_fsop_geom , + ) ->Result <WeakHandle <'a>> {Ok (WeakHandle { mountpoint, handle: xfs_handle::try_from (fp)?, + has_parent: fsgeom.has_parent (), }) } + +/// Create a new file handle from this one +pub fn subst (&self, fid:XfsFid ) ->xfs_handle { +xfs_handle { + ha_fid:xfs_fid { + fid_ino: fid.ino.into (), + fid_gen: fid.gen.into (), + ..self.handle.ha_fid + }, + ..self.handle + } + } + +/// Can this filesystem do parent pointer lookups? +pub fn can_get_parents (&self) ->bool { + self.has_parent + } }impl Display for WeakHandle <'_> {
PATCH 11/19 xfs_healer: make the rust program check if kernel support available
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Teach the rust port program to check if kernel support is available, to duplicate existing python functionality.
--- healer/src/healthmon/mod.rs | 28 ++++++++++++++++++++++++++++ healer/src/main.rs | 18 ++++++++++++++++++ 2 files changed, 46 insertions(+)
healer/src/healthmon/mod.rs +28 -0
@@ -3,8 +3,12 @@ * Copyright (C) 2025 Oracle. All Rights Reserved. * Author: Darrick J. Wong <[email protected]> */ +use crate::xfs_fs;use crate ::xfs_fs::xfs_health_monitor;use nix::ioctl_write_ptr; +use std::fs::File ; +use std::os::fd::AsRawFd ; +use std::os::fd::FromRawFd ;pub mod cstruct;pub mod event;
@@ -14,3 +18,27 @@ pub mod inodes;pub mod json;ioctl_write_ptr! (xfs_ioc_health_monitor,'X' ,68 , xfs_health_monitor); + +/// Check if the open file supports a health monitor. +pub fn is_supported (fp: &File , use_json:bool ) ->bool { +let hminfo =xfs_health_monitor { + format:if use_json { + xfs_fs::XFS_HEALTH_MONITOR_FMT_JSON as u8 + }else { + xfs_fs::XFS_HEALTH_MONITOR_FMT_CSTRUCT as u8 + }, + ..Default ::default () + }; + +// SAFETY: Trusting the kernel not to corrupt our memory, and for it to return a valid file +// description number, which we immediately convert to a File and drop to close the fd. +unsafe { +match xfs_ioc_health_monitor (fp.as_raw_fd (), &hminfo) { +Ok (x) => { +File ::from_raw_fd (x); +true + } +Err (_) =>false , + } + } +}
healer/src/main.rs +18 -0
@@ -69,6 +69,12 @@ impl Cli { .help(M_("Always repair corrupt metadata")) .action(ArgAction::SetTrue), ) + .arg ( +Arg ::new ("check" ) + .long ("check" ) + .help (M_ ("Check that health monitoring is supported" )) + .action (ArgAction ::SetTrue ), + ) .get_matches()) } }
@@ -81,6 +87,7 @@ struct App { everything: bool, json: bool, repair: bool, + check: bool, path: PathBuf, }
@@ -155,6 +162,16 @@ impl App { } } +// Now that we know that we can repair if the user wanted to, make sure that the kernel +// supports reporting events if that was as far as the user wanted us to go. +if self.check { +return Ok (if xfs_healer::healthmon::is_supported (&fp, self.json) { +ExitCode ::SUCCESS + }else { +ExitCode ::FAILURE + }); + } +let fh =WeakHandle ::try_new (&fp, &self.path, fsgeom) .with_context (||M_ ("Configuring filesystem handle" ))?;
@@ -187,6 +204,7 @@ impl From<Cli> for App { path: cli.0.get_one::<PathBuf>("path").unwrap().to_path_buf(), json: cli.0.get_flag("json"), repair: cli.0.get_flag("repair"), + check: cli.0.get_flag("check"), } } }
PATCH 12/19 xfs_healer: use the autofsck fsproperty to select mode
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make the rust xfs_healer program query the autofsck filesystem property to figure out which operating mode it should use.
--- healer/Cargo.toml.in | 1 healer/Makefile | 1 healer/src/fsprops.rs | 101 +++++++++++++++++++++++++++++++++++++++++++++++++ healer/src/lib.rs | 1 healer/src/main.rs | 93 +++++++++++++++++++++++++++++++++++++++++---- m4/package_rust.m4 | 1 6 files changed, 189 insertions(+), 9 deletions(-) create mode 100644 healer/src/fsprops.rs
healer/Cargo.toml.in +1 -0
@@ -28,6 +28,7 @@ serde_json = { version = "1.0.87" } nix = { version = "0", features = ["ioctl"] } # 0.26.1 strum = { version = "0" } # 0.19.2 strum_macros = { version = "0" } # 0.19.2 +libc = { version = "0" } # 0.2.139 # Dynamically comment out all the gettextrs related dependency information in # Cargo.toml becuse cargo requires the crate to be present so that it can
healer/Makefile +1 -0
@@ -18,6 +18,7 @@ HFILES = \ RUSTFILES = \ src/fsgeom.rs \ + src/fsprops.rs \ src/getparents.rs \ src/lib.rs \ src/main.rs \
healer/src/fsprops.rs new file +101 -0
101 lines not on this page · raw message
healer/src/lib.rs +1 -0
@@ -13,3 +13,4 @@ pub mod weakhandle;pub mod repair;pub mod fsgeom;pub mod getparents; +pub mod fsprops;
healer/src/main.rs +84 -9
155 lines not on this page · raw message
m4/package_rust.m4 +1 -0
@@ -134,6 +134,7 @@ enumset = { version = "1.0.12" } strum = { version = "0" } # 0.19.2 strum_macros = { version = "0" } # 0.19.2 serde_json = { version = "1.0.87" } +libc = { version = "0" } # 0.2.139 ], [yes], [no]) ])
PATCH 13/19 xfs_healer: use rc on the mountpoint instead of lifetime annotations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Institute a refcount on the mountpoint pathbuf so that we don't have to play around with explicit lifetime rules. That was fine for single threaded operation, but in the next patch we'll move the event processing to another thread to reduce time between read() calls.
--- healer/src/getparents.rs | 2 +- healer/src/main.rs | 9 +++++---- healer/src/weakhandle.rs | 19 ++++++++++--------- 3 files changed, 16 insertions(+), 14 deletions(-)
healer/src/getparents.rs +1 -1
@@ -180,7 +180,7 @@ fn find_path_components(Some (false ) } -impl WeakHandle <'_> { +impl WeakHandle {/// Return a path to the root for the given soft handle and ino/gen info, /// or None if errors occurred or we couldn't find the root. pub fn path_for (&self, fid:XfsFid ) ->Option <PathBuf > {
healer/src/main.rs +5 -4
@@ -8,6 +8,7 @@ use clap::{value_parser, Arg, ArgAction, ArgGroup, ArgMatches, Command};use std::fs::File ;use std::path::PathBuf ;use std::process::ExitCode ; +use std::rc::Rc ;use xfs_healer::fsprops;use xfs_healer::fsprops::XfsAutofsck ;use xfs_healer::healthmon::cstruct::CStructMonitor ;
@@ -98,7 +99,7 @@ struct App { repair: bool, check: bool, autofsck: bool, - path: PathBuf, + path: Rc<PathBuf>, }/// Outcome of checking if the kernel supports metadata repair
@@ -214,7 +215,7 @@ impl App {/// Main app method fn main (&mut self) ->Result <ExitCode > { -let fp =File ::open (&self.path).with_context (||M_ ("Opening filesystem failed" ))?; +let fp =File ::open (&*self.path).with_context (||M_ ("Opening filesystem failed" ))?;// Decide if we're going to enable repairs, which must come before check_repair. if self.autofsck {
@@ -246,7 +247,7 @@ impl App { }); } -let fh =WeakHandle ::try_new (&fp, &self.path, fsgeom) +let fh =WeakHandle ::try_new (&fp, self.path.clone (), fsgeom) .with_context (||M_ ("Configuring filesystem handle" ))?;if self.json {
@@ -275,7 +276,7 @@ impl From<Cli> for App { debug: cli.0.get_flag("debug"), log: cli.0.get_flag("log"), everything: cli.0.get_flag("everything"), - path: cli.0.get_one::<PathBuf>("path").unwrap().to_path_buf(), + path: Rc::new(cli.0.get_one::<PathBuf>("path").unwrap().to_path_buf()), json: cli.0.get_flag("json"), repair: cli.0.get_flag("repair"), check: cli.0.get_flag("check"),
healer/src/weakhandle.rs +10 -9
@@ -19,7 +19,8 @@ use std::fs::File;use std::io::ErrorKind ;use std::os::fd::AsRawFd ;use std::os::raw::c_void; -use std::path::Path ; +use std::path::PathBuf ; +use std::rc::Rc ;ioctl_readwrite! (xfs_ioc_fd_to_handle,'X' ,106 , xfs_fsop_handlereq);
@@ -70,9 +71,9 @@ impl TryFrom<&File> for xfs_handle { }/// Filesystem handle that can be disconnected from any open files -pub struct WeakHandle <'a> { +pub struct WeakHandle {/// path to the filesystem mountpoint - mountpoint: &'aPath , + mountpoint:Rc <PathBuf >,/// Filesystem handle handle:xfs_handle ,
@@ -81,10 +82,10 @@ pub struct WeakHandle<'a> { has_parent: bool, } -impl WeakHandle<'_> { +impl WeakHandle { /// Try to reopen the filesystem from which we got the handle. pub fn reopen(&self) -> Result<File> { - let fp = File::open(self.mountpoint)?; + let fp = File::open(self.mountpoint.as_path())?; if xfs_handle::try_from(&fp)? != self.handle { let s = format!(
@@ -105,11 +106,11 @@ impl WeakHandle<'_> { }/// Create a soft handle from an open file descriptor and its mount point -pub fn try_new <'a>( +pub fn try_new ( fp: &File , - mountpoint: &'aPath , + mountpoint:Rc <PathBuf >, fsgeom:xfs_fsop_geom , - ) ->Result <WeakHandle <'a>> { + ) ->Result <WeakHandle > {Ok (WeakHandle { mountpoint, handle: xfs_handle::try_from (fp)?,
@@ -135,7 +136,7 @@ impl WeakHandle<'_> { } } -impl Display for WeakHandle <'_> { +impl Display for WeakHandle {fn fmt (&self, f: &mut Formatter ) -> std::fmt::Result {write! (f,"{}" , self.mountpoint.display ()) }
PATCH 14/19 xfs_healer: use thread pools
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use a thread pool so that the kernel event reader thread can spend as much time sleeping in the kernel while other threads actually deal with decoding and processing the events.
--- healer/Cargo.toml.in | 1 + healer/src/main.rs | 87 +++++++++++++++++++++++++++++++++++++++------- healer/src/weakhandle.rs | 6 ++- m4/package_rust.m4 | 1 + 4 files changed, 78 insertions(+), 17 deletions(-)
healer/Cargo.toml.in +1 -0
@@ -18,6 +18,7 @@ clap = { version = "4.0.32", features = ["derive"] } anyhow = { version = "1.0.69" } enumset = { version = "1.0.12" } serde_json = { version = "1.0.87" } +threadpool = { version = "1.8.1" } # XXX: Crates with major version 0 are not considered ABI-stable, so the minor # version is treated as if it were the major version. This creates problems
healer/src/main.rs +73 -14
155 lines not on this page · raw message
healer/src/weakhandle.rs +3 -3
@@ -20,7 +20,7 @@ use std::io::ErrorKind;use std::os::fd::AsRawFd ;use std::os::raw::c_void;use std::path::PathBuf ; -use std::rc::Rc ; +use std::sync::Arc ;ioctl_readwrite! (xfs_ioc_fd_to_handle,'X' ,106 , xfs_fsop_handlereq);
@@ -73,7 +73,7 @@ impl TryFrom<&File> for xfs_handle {/// Filesystem handle that can be disconnected from any open files pub struct WeakHandle {/// path to the filesystem mountpoint - mountpoint:Rc <PathBuf >, + mountpoint:Arc <PathBuf >,/// Filesystem handle handle:xfs_handle ,
@@ -108,7 +108,7 @@ impl WeakHandle {/// Create a soft handle from an open file descriptor and its mount point pub fn try_new ( fp: &File , - mountpoint:Rc <PathBuf >, + mountpoint:Arc <PathBuf >, fsgeom:xfs_fsop_geom , ) ->Result <WeakHandle > {Ok (WeakHandle {
m4/package_rust.m4 +1 -0
@@ -135,6 +135,7 @@ strum = { version = "0" } # 0.19.2 strum_macros = { version = "0" } # 0.19.2 serde_json = { version = "1.0.87" } libc = { version = "0" } # 0.2.139 +threadpool = { version = "1.8.1" } ], [yes], [no]) ])
PATCH 15/19 xfs_healer: run full scrub after lost corruption events or targeted repair failure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If we fail to perform a spot repair of metadata or the kernel tells us that it lost corruption events due to queue limits, initiate a full run of the online fsck service to try to fix the error.
--- healer/Makefile | 1 + healer/src/healthmon/event.rs | 12 +++++++- healer/src/main.rs | 4 ++- healer/src/repair.rs | 60 ++++++++++++++++++++++++++++++++++++++++- healer/src/util.rs | 9 ++++++ healer/src/weakhandle.rs | 22 +++++++++++++++ healer/src/xfsprogs.rs.in | 1 + 7 files changed, 104 insertions(+), 5 deletions(-)
healer/Makefile +1 -0
@@ -131,6 +131,7 @@ src/xfsprogs.rs: src/xfsprogs.rs.in $(builddefs) $(Q)$(SED) -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@PACKAGE@|$(PKG_NAME)|g" \ -e "s|@LOCALEDIR@|$(PKG_LOCALE_DIR)|g" \ + -e "s|@scrub_svcname@|$(XFS_SCRUB_SVCNAME)|g" \ < $< > $@ $(CARGO_MANIFEST): $(CARGO_MANIFEST).in $(builddefs)
healer/src/healthmon/event.rs +10 -2
@@ -22,7 +22,7 @@ pub trait XfsHealthEvent {fn format (&self, fh: &WeakHandle ) -> (Option <PathBuf >,String );/// Generate the inputs to a kernel scrub ioctl -fn schedule_repairs (&self) ->Vec <Repair > { +fn schedule_repairs (&self, _everything:bool ) ->Vec <Repair > {vec! [] } }
@@ -32,7 +32,7 @@ pub trait XfsHealthEvent {#[macro_export] macro_rules! schedule_repairs { ($event_type:ty , $lambda: expr ) => { -fn schedule_repairs (&self) ->Vec <$crate::repair::Repair > { +fn schedule_repairs (&self, _:bool ) ->Vec <$crate::repair::Repair > {if self.status != $crate::healthmon::event::XfsHealthStatus ::Sick {return vec! []; }
@@ -89,6 +89,14 @@ impl XfsHealthEvent for LostEvent {fn format (&self, _: &WeakHandle ) -> (Option <PathBuf >,String ) { (None ,format! ("{} {}" , self.count,M_ ("events lost" ))) } + +fn schedule_repairs (&self, everything:bool ) ->Vec <Repair > { +if everything { +vec! [] + }else { +vec! [Repair ::full_repair ()] + } + } }/// Event for the monitor starting up
healer/src/main.rs +3 -1
@@ -109,6 +109,7 @@ struct App {#[derive( Debug )] struct EventThread { log:bool , + everything:bool , repair:bool , }
@@ -118,6 +119,7 @@ impl EventThread {fn new (app: &App ) ->Self {EventThread { log: app.log, + everything: app.everything, repair: app.repair, } }
@@ -162,7 +164,7 @@ impl App { }; }if et.repair { -for mut repairin event.schedule_repairs () { +for mut repairin event.schedule_repairs (et.everything) { repair.perform (&fh) } }
healer/src/repair.rs +58 -2
127 lines not on this page · raw message
healer/src/util.rs +9 -0
@@ -55,6 +55,15 @@ macro_rules! display_for_enum { }; } +/// Simple macro for creating errors for random badness. The only parameter describes why the +/// badness happened. +#[macro_export] +macro_rules! badness { + ($message:expr) => {{ + std::io::Error ::new (std::io::ErrorKind ::Other , $message) + }}; +} +/// Format an enum set into a string pub fn format_set <T :EnumSetType +Display >(f:EnumSet <T >) ->String {let mut ret ="" .to_string ();
healer/src/weakhandle.rs +22 -0
@@ -4,6 +4,7 @@ * Author: Darrick J. Wong <[email protected]> */ use crate::baddata; +use crate ::badness;use crate ::xfs_fs::xfs_fid;use crate ::xfs_fs::xfs_fsop_geom;use crate ::xfs_fs::xfs_fsop_handlereq;
@@ -13,13 +14,16 @@ use crate::xfsprogs::M_;use anyhow::{Error ,Result };use nix::ioctl_readwrite;use nix::libc::O_LARGEFILE ; +use std::ffi::OsString ;use std::fmt::Display ;use std::fmt::Formatter ;use std::fs::File ;use std::io::ErrorKind ;use std::os::fd::AsRawFd ;use std::os::raw::c_void; +use std::os::unix::ffi::OsStringExt ;use std::path::PathBuf ; +use std::process::Command ;use std::sync::Arc ;ioctl_readwrite! (xfs_ioc_fd_to_handle,'X' ,106 , xfs_fsop_handlereq);
@@ -134,6 +138,24 @@ impl WeakHandle {pub fn can_get_parents (&self) ->bool { self.has_parent } + +/// Compute the systemd instance unit name for this mountpoint. +pub fn instance_unit_name (&self, service_template: &str ) ->Result <OsString > { +let output =Command ::new ("systemd-escape" ) + .arg ("--template" ) + .arg (service_template) + .arg ("--path" ) + .arg (self.mountpoint.as_ref ()) + .output ()?; + +if !output.status.success () { +return Err (badness! ("Could not format systemd instance unit name." ).into ()); + } + +// systemd always adds a newline to the end of the output; remove it +let trunc_out = &output.stdout[0 ..output.stdout.len () -1 ]; +Ok (OsString ::from_vec (trunc_out.to_vec ())) + } }impl Display for WeakHandle {
healer/src/xfsprogs.rs.in +1 -0
@@ -6,6 +6,7 @@ #![allow(unexpected_cfgs)] pub const VERSION: &str = "@pkg_version@"; +pub const XFS_SCRUB_SVCNAME: &str = "@scrub_svcname@"; /// Try to initialize a localization library. Like the other xfsprogs utilities, we don't care /// if this fails.
PATCH 16/19 xfs_healer: use getmntent in Rust to find moved filesystems
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Wrap the libc getmntent function in an iterator. This enables xfs_healer to record the fsname (or fs spec) of the mountpoint that it's running against, and use that fsname to walk /proc/mounts to re-find the filesystem if the mount has moved elsewhere if it needs to open the fs to perform repairs.
--- healer/Makefile | 1 healer/src/getmntent.rs | 117 ++++++++++++++++++++++++++++++++++++++++++++++ healer/src/lib.rs | 1 healer/src/weakhandle.rs | 52 ++++++++++++++++++-- 4 files changed, 165 insertions(+), 6 deletions(-) create mode 100644 healer/src/getmntent.rs
healer/Makefile +1 -0
@@ -19,6 +19,7 @@ HFILES = \ RUSTFILES = \ src/fsgeom.rs \ src/fsprops.rs \ + src/getmntent.rs \ src/getparents.rs \ src/lib.rs \ src/main.rs \
healer/src/getmntent.rs new file +117 -0
117 lines not on this page · raw message
healer/src/lib.rs +1 -0
@@ -14,3 +14,4 @@ pub mod repair;pub mod fsgeom;pub mod getparents;pub mod fsprops; +pub mod getmntent;
healer/src/weakhandle.rs +46 -6
105 lines not on this page · raw message
PATCH 17/19 xfs_healer: validate that repair fds point to the monitored fs in Rust
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When xfs_healer reopens a mountpoint to perform a repair, it should validate that the opened fd points to a file on the same filesystem as the one being monitored.
--- healer/Makefile | 3 ++- healer/src/getparents.rs | 2 +- healer/src/healthmon/cstruct.rs | 15 +++++++++++++-- healer/src/healthmon/json.rs | 15 +++++++++++++-- healer/src/healthmon/mod.rs | 3 +++ healer/src/healthmon/samefs.rs | 33 +++++++++++++++++++++++++++++++++ healer/src/main.rs | 30 +++++++++++++++++++++++++----- healer/src/repair.rs | 11 ++++++----- healer/src/weakhandle.rs | 14 +++++++++----- 9 files changed, 105 insertions(+), 21 deletions(-) create mode 100644 healer/src/healthmon/samefs.rs
healer/Makefile +2 -1
@@ -35,7 +35,8 @@ RUSTFILES = \ src/healthmon/groups.rs \ src/healthmon/inodes.rs \ src/healthmon/json.rs \ - src/healthmon/mod.rs + src/healthmon/mod.rs \ + src/healthmon/samefs.rs BUILT_RUSTFILES = \ src/xfs_fs.rs \
healer/src/getparents.rs +1 -1
@@ -188,7 +188,7 @@ impl WeakHandle {return None ; } -let fp =match self.reopen () { +let fp =match self.reopen (|_|true ) {Err (_) =>return None ,Ok (x) => x, };
healer/src/healthmon/cstruct.rs +13 -2
@@ -17,6 +17,7 @@ use crate::healthmon::groups::{XfsPeragEvent, XfsPeragMetadata};use crate ::healthmon::groups::{XfsRtgroupEvent ,XfsRtgroupMetadata };use crate ::healthmon::inodes::{XfsFileIoErrorEvent ,XfsFileIoErrorType };use crate ::healthmon::inodes::{XfsInodeEvent ,XfsInodeMetadata }; +use crate ::healthmon::samefs::SameFs ;use crate ::healthmon::xfs_ioc_health_monitor;use crate ::xfs_fs;use crate ::xfs_fs::xfs_health_monitor;
@@ -33,6 +34,7 @@ use std::io::Read;use std::os::fd::AsRawFd ;use std::os::fd::FromRawFd ;use std::path::Path ; +use std::sync::Arc ;/// Boilerplate to stamp out functions to convert a u32 mask to an enumset /// of the given enum type.
@@ -74,6 +76,9 @@ pub struct CStructMonitor<'a> {/// path to the filesystem mountpoint mountpoint: &'a Path, + + /// object that repair threads use to check their reopened files against the monitored fs + samefs: Arc<SameFs>, }impl CStructMonitor <'_> {
@@ -90,17 +95,23 @@ impl CStructMonitor<'_> {// SAFETY: Trusting the kernel ioctl not to corrupt stack contents, and to return us a valid // file description number. -let health_fp =unsafe { +let (health_fp, samefs) =unsafe {let health_fd =xfs_ioc_health_monitor (fp.as_raw_fd (), &hminfo)?; -File ::from_raw_fd (health_fd) + (File ::from_raw_fd (health_fd),SameFs ::new (health_fd)) };drop (fp);Ok (CStructMonitor { objiter:BufReader ::new (health_fp), mountpoint, + samefs: samefs.into (), }) } + +/// Return an object that can be used to check reopened files +pub fn new_samefs (&self) ->Arc <SameFs > { + self.samefs.clone () + } }enum_from_field! (XfsHealthStatus , {
healer/src/healthmon/json.rs +13 -2
@@ -17,6 +17,7 @@ use crate::healthmon::groups::{XfsPeragEvent, XfsPeragMetadata};use crate ::healthmon::groups::{XfsRtgroupEvent ,XfsRtgroupMetadata };use crate ::healthmon::inodes::{XfsFileIoErrorEvent ,XfsFileIoErrorType };use crate ::healthmon::inodes::{XfsInodeEvent ,XfsInodeMetadata }; +use crate ::healthmon::samefs::SameFs ;use crate ::healthmon::xfs_ioc_health_monitor;use crate ::printlogln;use crate ::xfs_fs;
@@ -38,6 +39,7 @@ use std::os::fd::AsRawFd;use std::os::fd::FromRawFd ;use std::path::Path ;use std::str::FromStr ; +use std::sync::Arc ;/// Boilerplate to stamp out functions to convert json array to an enumset /// of the given enum type; or return an error with the given message.
@@ -106,6 +108,9 @@ pub struct JsonMonitor<'a> {/// are we debugging? debug: bool, + +/// object that repair threads use to check their reopened files against the monitored fs + samefs: Arc<SameFs>, }impl JsonMonitor <'_> {
@@ -127,9 +132,9 @@ impl JsonMonitor<'_> {// SAFETY: Trusting the kernel ioctl not to corrupt stack contents, and to return us a valid // file description number. -let health_fp =unsafe { +let (health_fp, samefs) =unsafe {let health_fd =xfs_ioc_health_monitor (fp.as_raw_fd (), &hminfo)?; -File ::from_raw_fd (health_fd) + (File ::from_raw_fd (health_fd),SameFs ::new (health_fd)) };drop (fp);
@@ -137,8 +142,14 @@ impl JsonMonitor<'_> { lineiter: BufReader::new(health_fp).lines(), mountpoint, debug, + samefs: samefs.into(), }) } + +/// Return an object that can be used to check reopened files +pub fn new_samefs (&self) ->Arc <SameFs > { + self.samefs.clone () + } }/// Raw health event, used to create the real objects
healer/src/healthmon/mod.rs +3 -0
@@ -5,6 +5,7 @@ */use crate ::xfs_fs;use crate ::xfs_fs::xfs_health_monitor; +use crate ::xfs_fs::xfs_health_samefs;use nix::ioctl_write_ptr;use std::fs::File ;use std::os::fd::AsRawFd ;
@@ -16,8 +17,10 @@ pub mod fs;pub mod groups;pub mod inodes;pub mod json; +pub mod samefs;ioctl_write_ptr! (xfs_ioc_health_monitor,'X' ,68 , xfs_health_monitor); +ioctl_write_ptr! (xfs_ioc_health_samefs,'X' ,69 , xfs_health_samefs);/// Check if the open file supports a health monitor. pub fn is_supported (fp: &File , use_json:bool ) ->bool {
healer/src/healthmon/samefs.rs new file +33 -0
@@ -0,0 +1,33 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright (C) 2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +use crate ::healthmon::xfs_ioc_health_samefs; +use crate ::xfs_fs::xfs_health_samefs; +use std::fs::File ; +use std::os::fd::AsRawFd ; +use std::os::raw::c_int; + +pub struct SameFs (c_int ); + +/// Predicate object that unsafely borrows the raw fd from a health monitor to check if a reopened +/// file is actually on the same fs. +impl SameFs { +/// Create a new predicate from the given raw file descriptor. Caller must ensure that the +/// fd is not closed before this object is destroyed. +pub fn new (fd:c_int ) ->SameFs { +SameFs (fd) + } + +/// Does this file point to the same filesystem as the health monitor? +pub fn is_same_fs (&self, fp: &File ) ->bool { +let hms =xfs_health_samefs { + fd: fp.as_raw_fd (), + ..Default ::default () + }; + +// any error means this isn't the same fs mount + !matches! (unsafe {xfs_ioc_health_samefs (self.0 , &hms) },Err (_e)) + } +}
healer/src/main.rs +25 -5
@@ -16,6 +16,7 @@ use xfs_healer::healthmon::cstruct::CStructMonitor;use xfs_healer::healthmon::event::XfsHealthEvent ;use xfs_healer::healthmon::json::JsonEventWrapper ;use xfs_healer::healthmon::json::JsonMonitor ; +use xfs_healer::healthmon::samefs::SameFs ;use xfs_healer::printlogln;use xfs_healer::repair::Repair ;use xfs_healer::weakhandle::WeakHandle ;
@@ -148,6 +149,7 @@ impl App {/// Handle a health event that has been decoded into real objects fn process_event ( et:EventThread , + samefs:Arc <SameFs >, fh:Arc <WeakHandle >, cooked:Result <Box <dyn XfsHealthEvent >>, ) {
@@ -165,7 +167,7 @@ impl App { }if et.repair {for mut repairin event.schedule_repairs (et.everything) { - repair.perform (&fh) + repair.perform (&samefs, &fh) } } }
@@ -176,22 +178,24 @@ impl App {fn dispatch_json_event ( threads: &ThreadPool , et:EventThread , + samefs:Arc <SameFs >, fh:Arc <WeakHandle >, raw_event:JsonEventWrapper , ) { threads.execute (move || { -App ::process_event (et, fh, raw_event.cook ()); +App ::process_event (et, samefs, fh, raw_event.cook ()); }) }fn dispatch_cstruct_event ( threads: &ThreadPool , et:EventThread , + samefs:Arc <SameFs >, fh:Arc <WeakHandle >, raw_event:xfs_health_monitor_event , ) { threads.execute (move || { -App ::process_event (et, fh, raw_event.cook ()); +App ::process_event (et, samefs, fh, raw_event.cook ()); }) }
@@ -308,24 +312,40 @@ impl App {if self.json {let hmon =JsonMonitor ::try_new (fp, &self.path, self.everything, self.debug) .with_context (||M_ ("Opening js health monitor file" ))?; +let samefs = hmon.new_samefs ();for raw_eventin hmon { -App ::dispatch_json_event (&threads,EventThread ::new (self), fh.clone (), raw_event); +App ::dispatch_json_event ( + &threads, +EventThread ::new (self), + samefs.clone (), + fh.clone (), + raw_event, + ); } + +// Prohibit hmon from leaving scope (and closing the health mon fd) before the worker +// threads have finished whatever they're doing. + threads.join (); }else {let hmon =CStructMonitor ::try_new (fp, &self.path, self.everything) .with_context (||M_ ("Opening health monitor file" ))?; +let samefs = hmon.new_samefs ();for raw_eventin hmon {App ::dispatch_cstruct_event ( &threads,EventThread ::new (self), + samefs.clone (), fh.clone (), raw_event, ); } + +// Prohibit hmon from leaving scope (and closing the health mon fd) before the worker +// threads have finished whatever they're doing. + threads.join (); } - threads.join ();Ok (ExitCode ::SUCCESS ) }
healer/src/repair.rs +6 -5
@@ -8,6 +8,7 @@ use crate::display_for_enum;use crate ::healthmon::fs::XfsWholeFsMetadata ;use crate ::healthmon::groups::{XfsPeragMetadata ,XfsRtgroupMetadata };use crate ::healthmon::inodes::XfsInodeMetadata ; +use crate ::healthmon::samefs::SameFs ;use crate ::printlogln;use crate ::weakhandle::WeakHandle ;use crate ::xfs_fs;
@@ -339,7 +340,7 @@ impl Repair { }/// Call the kernel to repair things -fn repair (&mut self, fh: &WeakHandle ) ->Result <bool > { +fn repair (&mut self, samefs: &SameFs , fh: &WeakHandle ) ->Result <bool > {if self.group ==RepairGroup ::FullRepair {let started = self .run_full_repair (fh)
@@ -351,7 +352,7 @@ impl Repair { }let fp = fh - .reopen () + .reopen (|fp| samefs.is_same_fs (fp)) .with_context (||M_ ("Reopening filesystem to repair metadata" ))?;// SAFETY: Trusting the kernel not to corrupt memory.
@@ -365,8 +366,8 @@ impl Repair { }/// Try to repair something, or log whatever went wrong -pub fn perform (&mut self, fh: &WeakHandle ) { -match self.repair (fh) { +pub fn perform (&mut self, samefs: &SameFs , fh: &WeakHandle ) { +match self.repair (samefs, fh) {Err (e) => {eprintln! ("{}: {:#}" , self.repair_path (fh), e); }
@@ -381,7 +382,7 @@ impl Repair {// Transform into a full repair if we failed to fix things. if self.outcome ==RepairOutcome ::Failed && self.group !=RepairGroup ::FullRepair { self.group =RepairGroup ::FullRepair ; - self.perform (fh); + self.perform (samefs, fh); } } };
healer/src/weakhandle.rs +9 -5
@@ -92,10 +92,14 @@ pub struct WeakHandle { impl WeakHandle { /// Try to reopen the filesystem with a given mountpoint - fn reopen_from(&self, mountpoint: &Path) -> Result<File> { + fn reopen_from( + &self, + mountpoint: &Path, + is_acceptable: impl Fn(&File) -> bool, + ) -> Result<File> { let fp = File::open(mountpoint)?; - if xfs_handle::try_from(&fp)? != self.handle { + if xfs_handle::try_from(&fp)? != self.handle || !is_acceptable(&fp) { let s = format!( "{} {}: {}", M_("reopening"),
@@ -109,9 +113,9 @@ impl WeakHandle { }/// Try to reopen the filesystem from which we got the handle. -pub fn reopen (&self) ->Result <File > { +pub fn reopen (&self, is_acceptable:impl Fn (&File ) ->bool ) ->Result <File > {// First try the original mountpoint -let orig_result = self.reopen_from (&self.mountpoint); +let orig_result = self.reopen_from (&self.mountpoint, &is_acceptable);if let Ok (x) = orig_result {return Ok (x); }
@@ -119,7 +123,7 @@ impl WeakHandle {// Now scan /proc/self/mounts for any other bind mounts of this filesystem let entries =MountEntries ::try_new ()?;for mntentin entries.filter (|x| x.fstype =="xfs" && x.fsname == self.fsname) { -if let Ok (x) = self.reopen_from (&mntent.dir) { +if let Ok (x) = self.reopen_from (&mntent.dir, &is_acceptable) {return Ok (x); } }
PATCH 18/19 debian/control: listify the build dependencies
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
This will make it less gross to add more build deps later.
--- debian/control | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
debian/control +13 -1
@@ -3,7 +3,19 @@ Section: admin Priority: optional Maintainer: XFS Development Team <[email protected]> Uploaders: Nathan Scott <[email protected]>, Anibal Monsalve Salazar <[email protected]>, Bastian Germann <[email protected]> -Build-Depends: libinih-dev (>= 53), uuid-dev, debhelper (>= 12), gettext, libtool, libedit-dev, libblkid-dev (>= 2.17), linux-libc-dev, libdevmapper-dev, libicu-dev, pkg-config, liburcu-dev, systemd-dev | systemd (<< 253-2~) +Build-Depends: debhelper (>= 12), + gettext, + libblkid-dev (>= 2.17), + libdevmapper-dev, + libedit-dev, + libicu-dev, + libinih-dev (>= 53), + libtool, + liburcu-dev, + linux-libc-dev, + pkg-config, + systemd-dev | systemd (<< 253-2~), + uuid-dev Standards-Version: 4.0.0 Homepage: https://xfs.wiki.kernel.org/
PATCH 19/19 debian/control: pull in build dependencies for xfs_healer
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Pull in the build dependencies for xfs_healer so that we can build the much nicer Rust version.
--- debian/control | 16 ++++++++++++++++ debian/rules | 1 + 2 files changed, 17 insertions(+)
debian/control +16 -0
@@ -4,16 +4,32 @@ Priority: optional Maintainer: XFS Development Team <[email protected]> Uploaders: Nathan Scott <[email protected]>, Anibal Monsalve Salazar <[email protected]>, Bastian Germann <[email protected]> Build-Depends: debhelper (>= 12), + cargo-web | cargo, + clang, gettext, libblkid-dev (>= 2.17), libdevmapper-dev, libedit-dev, libicu-dev, libinih-dev (>= 53), + librust-anyhow-dev, + librust-clap-derive-dev, + librust-clap-dev, + librust-derive-more-dev, + librust-enumset-derive-dev, + librust-enumset-dev, + librust-gettext-rs-dev, + librust-nix-dev, + librust-serde-json-dev, + librust-strum-dev, + librust-strum-macros-dev, + librust-threadpool-dev, libtool, liburcu-dev, linux-libc-dev, pkg-config, + rustc-web | rustc, + rustfmt-web | rustfmt, systemd-dev | systemd (<< 253-2~), uuid-dev Standards-Version: 4.0.0
debian/rules +1 -0
@@ -41,6 +41,7 @@ configure_options = \ --enable-lto \ --enable-release \ --with-system-crates \ + --disable-crate-checks \ --localstatedir=/var options = export DEBUG=-DNDEBUG DISTRIBUTION=debian \
Re: [PATCHSET V2 2/2] xfsprogs: autonomous self healing of filesystems in Rust
In reply to Darrick J. WongQuoted 8 lines · Darrick J. Wong
Hi all,
The initial implementation of the self healing daemon is written in Python. This was useful for rapid prototyping, but a more performant and typechecked codebase is valuable. Write a second implementation in Rust to reduce event processing overhead and library dependence. This could have been done in C, but I decided to use an environment with somewhat fewer footguns.
Having discarded the json output format last week, I decided to rewrite the Python version of xfs_healer in C partly out of curiosity and partly because I didn't see much advantage to having a Python script to call ioctls and interpret C structs. After removing the json support from the Rust version, the release binary sizes are:
-rwxr-xr-x root/root 1051096 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer
-rwxr-xr-x root/root 43904 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer.orig
This is a nearly 24x size increase to have Rust. I'm a n00b Rustacean and a veteran C stuckee, but between that and the difficulties of integrating two languages and two build systems together, I don't think it's worth the trouble to keep the Rust code. I've made a final push with the Rust code to my dev repo for the sake of posterity:
But I'm deleting this from my tree after I send this email.
That said, I quite enjoyed using this an excuse to familiarize myself with how to write bad Rust code. Using traits and the newtype pattern for geometric units (e.g. xfs_fsblock_t) was very helpful in keeping unit conversions understandable; and having to think about object access and lifetimes helped me produce a stable prototype very quickly. It also helps that rustc errors are far more helpful than gcc.
The only thing I didn't particularly like is the forced coordination for shared resources that already coordinate threads -- you can't easily have multiple readers sharing an open fd, even if that magic fd only emits struct sized objects and takes i_rwsem exclusively to prevent corruption problems.
Dealing with cargo for a distro package build was nightmarish -- hermetically sealed build systems (you want this) can't access crates.io which means that I as the author had to be careful only to use crate packages that are in EPEL or Debian stable, and to tell cargo only to look on the local filesystem. So I guess I now have experience in that, should anyone want to know how to do that.
(Also, how do you do i18n in Rust programs? gettext???)
--D
Quoted 90 lines
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring-rust
---
Commits in this patchset:
*xfs_healer: start building a Rust version
*xfs_healer: enable gettext for localization
*xfs_healer: bindgenxfs_fs.h
*xfs_healer: define Rust objects for health events and kernel interface
*xfs_healer: read binary health events from the kernel
*xfs_healer: read json health events from the kernel
*xfs_healer: create a weak file handle so we don't pin the mount
*xfs_healer: fix broken filesystem metadata
*xfs_healer: check for fs features needed for effective repairs
*xfs_healer: use getparents to look up file names
*xfs_healer: make the rust program check if kernel support available
*xfs_healer: use the autofsck fsproperty to select mode
*xfs_healer: use rc on the mountpoint instead of lifetime annotations
*xfs_healer: use thread pools
*xfs_healer: run full scrub after lost corruption events or targeted repair failure
*xfs_healer: use getmntent in Rust to find moved filesystems
*xfs_healer: validate that repair fds point to the monitored fs in Rust
* debian/control: listify the build dependencies
* debian/control: pull in build dependencies forxfs_healer--- healer/bindgen_xfs_fs.h | 6 + configure.ac | 84 ++++++++ debian/control | 30 +++ debian/rules | 3 healer/.cargo/config.toml.system | 6 + healer/Cargo.toml.in | 37 +++ healer/Makefile | 143 +++++++++++++ healer/rbindgen | 57 +++++ healer/src/fsgeom.rs | 41 ++++ healer/src/fsprops.rs | 101 +++++++++ healer/src/getmntent.rs | 117 +++++++++++ healer/src/getparents.rs | 210 ++++++++++++++++++++ healer/src/healthmon/cstruct.rs | 354 +++++++++++++++++++++++++++++++++ healer/src/healthmon/event.rs | 122 +++++++++++ healer/src/healthmon/fs.rs | 163 +++++++++++++++ healer/src/healthmon/groups.rs | 160 +++++++++++++++ healer/src/healthmon/inodes.rs | 142 +++++++++++++ healer/src/healthmon/json.rs | 409 ++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 47 ++++ healer/src/healthmon/samefs.rs | 33 +++ healer/src/lib.rs | 17 ++ healer/src/main.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/repair.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/util.rs | 81 ++++++++ healer/src/weakhandle.rs | 209 +++++++++++++++++++ healer/src/xfs_types.rs | 292 +++++++++++++++++++++++++++ healer/src/xfsprogs.rs.in | 33 +++ include/builddefs.in | 13 + include/buildrules | 1 m4/Makefile | 1 m4/package_rust.m4 | 163 +++++++++++++++ 31 files changed, 3851 insertions(+), 4 deletions(-) create mode 100644 healer/bindgen_xfs_fs.h create mode 100644 healer/.cargo/config.toml.system create mode 100644 healer/Cargo.toml.in create mode 100755 healer/rbindgen create mode 100644 healer/src/fsgeom.rs create mode 100644 healer/src/fsprops.rs create mode 100644 healer/src/getmntent.rs create mode 100644 healer/src/getparents.rs create mode 100644 healer/src/healthmon/cstruct.rs create mode 100644 healer/src/healthmon/event.rs create mode 100644 healer/src/healthmon/fs.rs create mode 100644 healer/src/healthmon/groups.rs create mode 100644 healer/src/healthmon/inodes.rs create mode 100644 healer/src/healthmon/json.rs create mode 100644 healer/src/healthmon/mod.rs create mode 100644 healer/src/healthmon/samefs.rs create mode 100644 healer/src/lib.rs create mode 100644 healer/src/main.rs create mode 100644 healer/src/repair.rs create mode 100644 healer/src/util.rs create mode 100644 healer/src/weakhandle.rs create mode 100644 healer/src/xfs_types.rs create mode 100644 healer/src/xfsprogs.rs.in create mode 100644 m4/package_rust.m4
Re: [PATCHSET V2 2/2] xfsprogs: autonomous self healing of filesystems in Rust
Quoted 19 lines · Darrick J. Wong
Quoted 8 lines · Darrick J. Wong
Hi all,
The initial implementation of the self healing daemon is written in Python. This was useful for rapid prototyping, but a more performant and typechecked codebase is valuable. Write a second implementation in Rust to reduce event processing overhead and library dependence. This could have been done in C, but I decided to use an environment with somewhat fewer footguns.
Having discarded the json output format last week, I decided to rewrite the Python version of
xfs_healerin C partly out of curiosity and partly because I didn't see much advantage to having a Python script to call ioctls and interpret C structs. After removing the json support from the Rust version, the release binary sizes are:(quoted diff) +0
-2-rwxr-xr-x root/root 1051096 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer -rwxr-xr-x root/root 43904 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer.orig
Quoted 1 line
This is a nearly 24x size increase to have Rust. I'm a n00b Rustacean
cargo build --release? (optimized + no debug info)
Quoted 8 lines
and a veteran C stuckee, but between that and the difficulties of integrating two languages and two build systems together, I don't think it's worth the trouble to keep the Rust code. I've made a final push with the Rust code to my dev repo for the sake of posterity:
But I'm deleting this from my tree after I send this email.
:'(
Quoted 116 lines
That said, I quite enjoyed using this an excuse to familiarize myself with how to write bad Rust code. Using traits and the newtype pattern for geometric units (e.g.
xfs_fsblock_t) was very helpful in keeping unit conversions understandable; and having to think about object access and lifetimes helped me produce a stable prototype very quickly. It also helps that rustc errors are far more helpful than gcc.The only thing I didn't particularly like is the forced coordination for shared resources that already coordinate threads -- you can't easily have multiple readers sharing an open fd, even if that magic fd only emits struct sized objects and takes
i_rwsemexclusively to prevent corruption problems.Dealing with cargo for a distro package build was nightmarish -- hermetically sealed build systems (you want this) can't access crates.io which means that I as the author had to be careful only to use crate packages that are in EPEL or Debian stable, and to tell cargo only to look on the local filesystem. So I guess I now have experience in that, should anyone want to know how to do that.
(Also, how do you do i18n in Rust programs? gettext???)
--D
Quoted 90 lines
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring-rust
---
Commits in this patchset:
*xfs_healer: start building a Rust version
*xfs_healer: enable gettext for localization
*xfs_healer: bindgenxfs_fs.h
*xfs_healer: define Rust objects for health events and kernel interface
*xfs_healer: read binary health events from the kernel
*xfs_healer: read json health events from the kernel
*xfs_healer: create a weak file handle so we don't pin the mount
*xfs_healer: fix broken filesystem metadata
*xfs_healer: check for fs features needed for effective repairs
*xfs_healer: use getparents to look up file names
*xfs_healer: make the rust program check if kernel support available
*xfs_healer: use the autofsck fsproperty to select mode
*xfs_healer: use rc on the mountpoint instead of lifetime annotations
*xfs_healer: use thread pools
*xfs_healer: run full scrub after lost corruption events or targeted repair failure
*xfs_healer: use getmntent in Rust to find moved filesystems
*xfs_healer: validate that repair fds point to the monitored fs in Rust
* debian/control: listify the build dependencies
* debian/control: pull in build dependencies forxfs_healer--- healer/bindgen_xfs_fs.h | 6 + configure.ac | 84 ++++++++ debian/control | 30 +++ debian/rules | 3 healer/.cargo/config.toml.system | 6 + healer/Cargo.toml.in | 37 +++ healer/Makefile | 143 +++++++++++++ healer/rbindgen | 57 +++++ healer/src/fsgeom.rs | 41 ++++ healer/src/fsprops.rs | 101 +++++++++ healer/src/getmntent.rs | 117 +++++++++++ healer/src/getparents.rs | 210 ++++++++++++++++++++ healer/src/healthmon/cstruct.rs | 354 +++++++++++++++++++++++++++++++++ healer/src/healthmon/event.rs | 122 +++++++++++ healer/src/healthmon/fs.rs | 163 +++++++++++++++ healer/src/healthmon/groups.rs | 160 +++++++++++++++ healer/src/healthmon/inodes.rs | 142 +++++++++++++ healer/src/healthmon/json.rs | 409 ++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 47 ++++ healer/src/healthmon/samefs.rs | 33 +++ healer/src/lib.rs | 17 ++ healer/src/main.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/repair.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/util.rs | 81 ++++++++ healer/src/weakhandle.rs | 209 +++++++++++++++++++ healer/src/xfs_types.rs | 292 +++++++++++++++++++++++++++ healer/src/xfsprogs.rs.in | 33 +++ include/builddefs.in | 13 + include/buildrules | 1 m4/Makefile | 1 m4/package_rust.m4 | 163 +++++++++++++++ 31 files changed, 3851 insertions(+), 4 deletions(-) create mode 100644 healer/bindgen_xfs_fs.h create mode 100644 healer/.cargo/config.toml.system create mode 100644 healer/Cargo.toml.in create mode 100755 healer/rbindgen create mode 100644 healer/src/fsgeom.rs create mode 100644 healer/src/fsprops.rs create mode 100644 healer/src/getmntent.rs create mode 100644 healer/src/getparents.rs create mode 100644 healer/src/healthmon/cstruct.rs create mode 100644 healer/src/healthmon/event.rs create mode 100644 healer/src/healthmon/fs.rs create mode 100644 healer/src/healthmon/groups.rs create mode 100644 healer/src/healthmon/inodes.rs create mode 100644 healer/src/healthmon/json.rs create mode 100644 healer/src/healthmon/mod.rs create mode 100644 healer/src/healthmon/samefs.rs create mode 100644 healer/src/lib.rs create mode 100644 healer/src/main.rs create mode 100644 healer/src/repair.rs create mode 100644 healer/src/util.rs create mode 100644 healer/src/weakhandle.rs create mode 100644 healer/src/xfs_types.rs create mode 100644 healer/src/xfsprogs.rs.in create mode 100644 m4/package_rust.m4
Signature · 1 line
- Andrey
Re: [PATCHSET V2 2/2] xfsprogs: autonomous self healing of filesystems in Rust
Quoted 17 lines · Andrey Albershteyn
Quoted 16 lines · Darrick J. Wong
Quoted 8 lines · Darrick J. Wong
Hi all,
The initial implementation of the self healing daemon is written in Python. This was useful for rapid prototyping, but a more performant and typechecked codebase is valuable. Write a second implementation in Rust to reduce event processing overhead and library dependence. This could have been done in C, but I decided to use an environment with somewhat fewer footguns.
Having discarded the json output format last week, I decided to rewrite the Python version of
xfs_healerin C partly out of curiosity and partly because I didn't see much advantage to having a Python script to call ioctls and interpret C structs. After removing the json support from the Rust version, the release binary sizes are:
Quoted 6 lines
Quoted 4 lines
(quoted diff) +0
-2-rwxr-xr-x root/root 1051096 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer -rwxr-xr-x root/root 43904 2025-11-04 14:25 ./usr/libexec/xfsprogs/xfs_healer.origThis is a nearly 24x size increase to have Rust. I'm a n00b Rustacean
cargo build --release? (optimized + no debug info)
That (1.05M) is the release build size. The debug build size was 33MB. Not sure how or why Rust binaries get so huge, but there it is. :/
--D
Quoted 132 lines
Quoted 8 lines
and a veteran C stuckee, but between that and the difficulties of integrating two languages and two build systems together, I don't think it's worth the trouble to keep the Rust code. I've made a final push with the Rust code to my dev repo for the sake of posterity:
But I'm deleting this from my tree after I send this email.
:'(
Quoted 116 lines
That said, I quite enjoyed using this an excuse to familiarize myself with how to write bad Rust code. Using traits and the newtype pattern for geometric units (e.g.
xfs_fsblock_t) was very helpful in keeping unit conversions understandable; and having to think about object access and lifetimes helped me produce a stable prototype very quickly. It also helps that rustc errors are far more helpful than gcc.The only thing I didn't particularly like is the forced coordination for shared resources that already coordinate threads -- you can't easily have multiple readers sharing an open fd, even if that magic fd only emits struct sized objects and takes
i_rwsemexclusively to prevent corruption problems.Dealing with cargo for a distro package build was nightmarish -- hermetically sealed build systems (you want this) can't access crates.io which means that I as the author had to be careful only to use crate packages that are in EPEL or Debian stable, and to tell cargo only to look on the local filesystem. So I guess I now have experience in that, should anyone want to know how to do that.
(Also, how do you do i18n in Rust programs? gettext???)
--D
Quoted 90 lines
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring-rust
---
Commits in this patchset:
*xfs_healer: start building a Rust version
*xfs_healer: enable gettext for localization
*xfs_healer: bindgenxfs_fs.h
*xfs_healer: define Rust objects for health events and kernel interface
*xfs_healer: read binary health events from the kernel
*xfs_healer: read json health events from the kernel
*xfs_healer: create a weak file handle so we don't pin the mount
*xfs_healer: fix broken filesystem metadata
*xfs_healer: check for fs features needed for effective repairs
*xfs_healer: use getparents to look up file names
*xfs_healer: make the rust program check if kernel support available
*xfs_healer: use the autofsck fsproperty to select mode
*xfs_healer: use rc on the mountpoint instead of lifetime annotations
*xfs_healer: use thread pools
*xfs_healer: run full scrub after lost corruption events or targeted repair failure
*xfs_healer: use getmntent in Rust to find moved filesystems
*xfs_healer: validate that repair fds point to the monitored fs in Rust
* debian/control: listify the build dependencies
* debian/control: pull in build dependencies forxfs_healer--- healer/bindgen_xfs_fs.h | 6 + configure.ac | 84 ++++++++ debian/control | 30 +++ debian/rules | 3 healer/.cargo/config.toml.system | 6 + healer/Cargo.toml.in | 37 +++ healer/Makefile | 143 +++++++++++++ healer/rbindgen | 57 +++++ healer/src/fsgeom.rs | 41 ++++ healer/src/fsprops.rs | 101 +++++++++ healer/src/getmntent.rs | 117 +++++++++++ healer/src/getparents.rs | 210 ++++++++++++++++++++ healer/src/healthmon/cstruct.rs | 354 +++++++++++++++++++++++++++++++++ healer/src/healthmon/event.rs | 122 +++++++++++ healer/src/healthmon/fs.rs | 163 +++++++++++++++ healer/src/healthmon/groups.rs | 160 +++++++++++++++ healer/src/healthmon/inodes.rs | 142 +++++++++++++ healer/src/healthmon/json.rs | 409 ++++++++++++++++++++++++++++++++++++++ healer/src/healthmon/mod.rs | 47 ++++ healer/src/healthmon/samefs.rs | 33 +++ healer/src/lib.rs | 17 ++ healer/src/main.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/repair.rs | 390 ++++++++++++++++++++++++++++++++++++ healer/src/util.rs | 81 ++++++++ healer/src/weakhandle.rs | 209 +++++++++++++++++++ healer/src/xfs_types.rs | 292 +++++++++++++++++++++++++++ healer/src/xfsprogs.rs.in | 33 +++ include/builddefs.in | 13 + include/buildrules | 1 m4/Makefile | 1 m4/package_rust.m4 | 163 +++++++++++++++ 31 files changed, 3851 insertions(+), 4 deletions(-) create mode 100644 healer/bindgen_xfs_fs.h create mode 100644 healer/.cargo/config.toml.system create mode 100644 healer/Cargo.toml.in create mode 100755 healer/rbindgen create mode 100644 healer/src/fsgeom.rs create mode 100644 healer/src/fsprops.rs create mode 100644 healer/src/getmntent.rs create mode 100644 healer/src/getparents.rs create mode 100644 healer/src/healthmon/cstruct.rs create mode 100644 healer/src/healthmon/event.rs create mode 100644 healer/src/healthmon/fs.rs create mode 100644 healer/src/healthmon/groups.rs create mode 100644 healer/src/healthmon/inodes.rs create mode 100644 healer/src/healthmon/json.rs create mode 100644 healer/src/healthmon/mod.rs create mode 100644 healer/src/healthmon/samefs.rs create mode 100644 healer/src/lib.rs create mode 100644 healer/src/main.rs create mode 100644 healer/src/repair.rs create mode 100644 healer/src/util.rs create mode 100644 healer/src/weakhandle.rs create mode 100644 healer/src/xfs_types.rs create mode 100644 healer/src/xfsprogs.rs.in create mode 100644 m4/package_rust.m4Signature · 1 line
- Andrey
PATCH v2 1/1 fstests: autonomous self healing of filesystems
In reply to Darrick J. WongHi all,
This series adds functionality and regression tests for the automated self healing daemon for xfs.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* xfs: test health monitoring code
* xfs: test for metadata corruption error reporting via healthmon
* xfs: test io error reporting via healthmon
* xfs: test new xfs_healer daemon
--- common/config | 6 + common/rc | 15 ++++ common/systemd | 21 +++++ common/xfs | 86 +++++++++++++++++++++ doc/group-names.txt | 1 tests/xfs/1878 | 80 ++++++++++++++++++++ tests/xfs/1878.out | 10 ++ tests/xfs/1879 | 89 ++++++++++++++++++++++ tests/xfs/1879.out | 12 +++ tests/xfs/1882 | 48 ++++++++++++ tests/xfs/1882.out | 2 tests/xfs/1883 | 60 +++++++++++++++ tests/xfs/1883.out | 2 tests/xfs/1884 | 90 ++++++++++++++++++++++ tests/xfs/1884.out | 2 tests/xfs/1885 | 53 +++++++++++++ tests/xfs/1885.out | 5 + tests/xfs/1896 | 206 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1896.out | 21 +++++ tests/xfs/1897 | 98 ++++++++++++++++++++++++ tests/xfs/1897.out | 4 + tests/xfs/1898 | 36 +++++++++ tests/xfs/1898.out | 4 + tests/xfs/1899 | 109 +++++++++++++++++++++++++++ tests/xfs/1899.out | 3 + tests/xfs/1900 | 116 +++++++++++++++++++++++++++++ tests/xfs/1900.out | 2 tests/xfs/1901 | 138 ++++++++++++++++++++++++++++++++++ tests/xfs/1901.out | 2 29 files changed, 1321 insertions(+) create mode 100755 tests/xfs/1878 create mode 100644 tests/xfs/1878.out create mode 100755 tests/xfs/1879 create mode 100644 tests/xfs/1879.out create mode 100755 tests/xfs/1882 create mode 100644 tests/xfs/1882.out create mode 100755 tests/xfs/1883 create mode 100644 tests/xfs/1883.out create mode 100755 tests/xfs/1884 create mode 100644 tests/xfs/1884.out create mode 100755 tests/xfs/1885 create mode 100644 tests/xfs/1885.out create mode 100755 tests/xfs/1896 create mode 100644 tests/xfs/1896.out create mode 100755 tests/xfs/1897 create mode 100755 tests/xfs/1897.out create mode 100755 tests/xfs/1898 create mode 100755 tests/xfs/1898.out create mode 100755 tests/xfs/1899 create mode 100644 tests/xfs/1899.out create mode 100755 tests/xfs/1900 create mode 100755 tests/xfs/1900.out create mode 100755 tests/xfs/1901 create mode 100755 tests/xfs/1901.out
PATCH 1/4 xfs: test health monitoring code
From: Darrick J. Wong <[email protected]>
Add some functionality tests for the new health monitoring code.
--- doc/group-names.txt | 1 + tests/xfs/1885 | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1885.out | 5 +++++ 3 files changed, 59 insertions(+) create mode 100755 tests/xfs/1885 create mode 100644 tests/xfs/1885.out
doc/group-names.txt +1 -0
@@ -117,6 +117,7 @@ samefs overlayfs when all layers are on the same fs scrub filesystem metadata scrubbers seed btrfs seeded filesystems seek llseek functionality +selfhealing self healing filesystem code selftest tests with fixed results, used to validate testing setup send btrfs send/receive shrinkfs decreasing the size of a filesystem
tests/xfs/1885 new file +53 -0
53 lines not on this page · raw message
tests/xfs/1885.out new file +5 -0
@@ -0,0 +1,5 @@ +QA output created by 1885 +mount refcount is in range +mount + healthmon refcount is in range +healthmon refcount is in range +end refcount is in range
PATCH 2/4 xfs: test for metadata corruption error reporting via healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Check if we can detect runtime metadata corruptions via the health monitor.
--- common/rc | 10 ++++++ tests/xfs/1879 | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1879.out | 12 +++++++ 3 files changed, 111 insertions(+) create mode 100755 tests/xfs/1879 create mode 100644 tests/xfs/1879.out
common/rc +10 -0
@@ -3037,6 +3037,16 @@ _require_xfs_io_command() echo $testio | grep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command support is missing" ;; + "healthmon") + testio=`$XFS_IO_PROG -c "$command -p $param" $TEST_DIR 2>&1` + echo $testio | grep -q "bad argument count" && \ + _notrun "xfs_io $command $param support is missing" + echo $testio | grep -q "Inappropriate ioctl" && \ + _notrun "xfs_io $command $param ioctl support is missing" + echo $testio | grep -q "Operation not supported" && \ + _notrun "xfs_io $command $param kernel support is missing" + param_checked="$param" + ;; "label") testio=`$XFS_IO_PROG -c "label" $TEST_DIR 2>&1` ;;
tests/xfs/1879 new file +89 -0
89 lines not on this page · raw message
tests/xfs/1879.out new file +12 -0
@@ -0,0 +1,12 @@ +QA output created by 1879 +Format and mount +Runtime corruption detection + "domain": "inode", + "structures": ["directory"], + "type": "sick", +Scrub corruption detection + "domain": "inode", + "structures": ["directory"], + "structures": ["parent"], + "type": "corrupt", + "type": "sick",
PATCH 3/4 xfs: test io error reporting via healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new test to make sure the kernel can report IO errors via health monitoring.
--- tests/xfs/1878 | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1878.out | 10 +++++++ 2 files changed, 90 insertions(+) create mode 100755 tests/xfs/1878 create mode 100644 tests/xfs/1878.out
tests/xfs/1878 new file +80 -0
@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test No. 1878 +# +# Attempt to read and write a file in buffered and directio mode with the +# health monitoring program running. Check that healthmon observes all four +# types of IO errors. +# +. ./common/preamble +_begin_fstest auto quick eio selfhealing + +_cleanup() +{ + cd / + rm -rf $tmp.* $testdir + _dmerror_cleanup +} + +. ./common/filter +. ./common/dmerror + +_require_scratch_nocheck +_require_xfs_io_command healthmon +_require_dm_target error + +# Disable the scratch rt device to avoid test failures relating to the rt +# bitmap consuming all the free space in our small data device. +unset SCRATCH_RTDEV + +echo "Format and mount" +_scratch_mkfs > $seqres.full 2>&1 +_dmerror_init no_log +_dmerror_mount + +_require_fs_space $SCRATCH_MNT 65536 + +# Create a file with written regions far enough apart that the pagecache can't +# possibly be caching the regions with a single folio. +testfile=$SCRATCH_MNT/fsync-err-test +$XFS_IO_PROG -f \ + -c 'pwrite -b 1m 0 1m' \ + -c 'pwrite -b 1m 10g 1m' \ + -c 'pwrite -b 1m 20g 1m' \ + -c fsync $testfile >> $seqres.full + +# First we check if directio errors get reported +$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT >> $tmp.healthmon & +sleep 1 # wait for python program to start up +_dmerror_load_error_table +$XFS_IO_PROG -d -c 'pwrite -b 256k 12k 16k' $testfile >> $seqres.full +$XFS_IO_PROG -d -c 'pread -b 256k 10g 16k' $testfile >> $seqres.full +_dmerror_load_working_table + +_dmerror_unmount +wait # for healthmon to finish +_dmerror_mount + +# Next we check if buffered io errors get reported. We have to write something +# before loading the error table to ensure the dquots get loaded. +$XFS_IO_PROG -c 'pwrite -b 256k 20g 1k' -c fsync $testfile >> $seqres.full +$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT >> $tmp.healthmon & +sleep 1 # wait for python program to start up +_dmerror_load_error_table +$XFS_IO_PROG -c 'pread -b 256k 12k 16k' $testfile >> $seqres.full +$XFS_IO_PROG -c 'pwrite -b 256k 20g 16k' -c fsync $testfile >> $seqres.full +_dmerror_load_working_table + +_dmerror_unmount +wait # for healthmon to finish + +# Did we get errors? +cat $tmp.healthmon >> $seqres.full +grep -E '(directio|readahead|writeback)' $tmp.healthmon | sort | uniq + +_dmerror_cleanup + +status=0 +exit
tests/xfs/1878.out new file +10 -0
@@ -0,0 +1,10 @@ +QA output created by 1878 +Format and mount +pwrite: Input/output error +pread: Input/output error +pread: Input/output error +fsync: Input/output error + "type": "directio_read", + "type": "directio_write", + "type": "readahead", + "type": "writeback",
PATCH 4/4 xfs: test new xfs_healer daemon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make sure the daemon in charge of self healing xfs actually does what it says it does -- emits json blobs that can be validated against the schema, repairs metadata, logs file IO errors, and doesn't get overwhelmed by event floods.
--- common/config | 6 ++ common/rc | 5 + common/systemd | 21 +++++ common/xfs | 86 ++++++++++++++++++++++ tests/xfs/1882 | 48 ++++++++++++ tests/xfs/1882.out | 2 + tests/xfs/1883 | 60 +++++++++++++++ tests/xfs/1883.out | 2 + tests/xfs/1884 | 90 +++++++++++++++++++++++ tests/xfs/1884.out | 2 + tests/xfs/1896 | 206 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1896.out | 21 +++++ tests/xfs/1897 | 98 +++++++++++++++++++++++++ tests/xfs/1897.out | 4 + tests/xfs/1898 | 36 +++++++++ tests/xfs/1898.out | 4 + tests/xfs/1899 | 109 ++++++++++++++++++++++++++++ tests/xfs/1899.out | 3 + tests/xfs/1900 | 116 +++++++++++++++++++++++++++++ tests/xfs/1900.out | 2 + tests/xfs/1901 | 138 +++++++++++++++++++++++++++++++++++ tests/xfs/1901.out | 2 + 22 files changed, 1061 insertions(+) create mode 100755 tests/xfs/1882 create mode 100644 tests/xfs/1882.out create mode 100755 tests/xfs/1883 create mode 100644 tests/xfs/1883.out create mode 100755 tests/xfs/1884 create mode 100644 tests/xfs/1884.out create mode 100755 tests/xfs/1896 create mode 100644 tests/xfs/1896.out create mode 100755 tests/xfs/1897 create mode 100755 tests/xfs/1897.out create mode 100755 tests/xfs/1898 create mode 100755 tests/xfs/1898.out create mode 100755 tests/xfs/1899 create mode 100644 tests/xfs/1899.out create mode 100755 tests/xfs/1900 create mode 100755 tests/xfs/1900.out create mode 100755 tests/xfs/1901 create mode 100755 tests/xfs/1901.out
common/config +6 -0
@@ -161,6 +161,12 @@ export XFS_ADMIN_PROG="$(type -P xfs_admin)" export XFS_GROWFS_PROG=$(type -P xfs_growfs) export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)" export XFS_SCRUB_PROG="$(type -P xfs_scrub)" +XFS_HEALER_PROG="$(type -P xfs_healer)" +# If the healer daemon isn't in the PATH, try the one installed in libexec +if [ -z "$XFS_HEALER_PROG" ] && [ -e /usr/libexec/xfs_healer ]; then + XFS_HEALER_PROG=/usr/libexec/xfs_healer +fi +export XFS_HEALER_PROG export XFS_PARALLEL_REPAIR_PROG="$(type -P xfs_prepair)" export XFS_PARALLEL_REPAIR64_PROG="$(type -P xfs_prepair64)" export __XFSDUMP_PROG="$(type -P xfsdump)"
common/rc +5 -0
@@ -3050,6 +3050,11 @@ _require_xfs_io_command() "label") testio=`$XFS_IO_PROG -c "label" $TEST_DIR 2>&1` ;; + "mediaerror") + testio=`$XFS_IO_PROG -x -c "mediaerror $* 0 0" 2>&1` + echo $testio | grep -q "invalid option" && \ + _notrun "xfs_io $command support is missing" + ;; "open") # -c "open $f" is broken in xfs_io <= 4.8. Along with the fix, # a new -C flag was introduced to execute one shot commands.
common/systemd +21 -0
@@ -44,6 +44,18 @@ _systemd_unit_active() { test "$(systemctl is-active "$1")" = "active" } +# Wait for up to a certain number of seconds for a service to reach inactive +# state. +_systemd_unit_wait() { + local svcname="$1" + local timeout="${2:-30}" + + for ((i = 0; i < (timeout * 2); i++)); do + test "$(systemctl is-active "$svcname")" = "inactive" && break + sleep 0.5 + done +} + _require_systemd_unit_active() { _require_systemd_unit_defined "$1" _systemd_unit_active "$1" || \
@@ -71,3 +83,12 @@ _systemd_unit_status() { _systemd_installed || return 1 systemctl status "$1" } + +# Start a running systemd unit +_systemd_unit_start() { + systemctl start "$1" +} +# Stop a running systemd unit +_systemd_unit_stop() { + systemctl stop "$1" +}
common/xfs +86 -0
89 lines not on this page · raw message
tests/xfs/1882 new file +48 -0
48 lines not on this page · raw message
tests/xfs/1882.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1882 +Silence is golden
tests/xfs/1883 new file +60 -0
60 lines not on this page · raw message
tests/xfs/1883.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1883 +Silence is golden
tests/xfs/1884 new file +90 -0
90 lines not on this page · raw message
tests/xfs/1884.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1884 +ls: reading directory 'SCRATCH_MNT/some/victimdir': Structure needs cleaning
tests/xfs/1896 new file +206 -0
206 lines not on this page · raw message
tests/xfs/1896.out new file +21 -0
@@ -0,0 +1,21 @@ +QA output created by 1896 +Try buffered read +pread: Input/output error +Try directio read +pread: Input/output error +Try directio write +pwrite: Input/output error +Try buffered write +fsync: Input/output error +Fix device +Kill healer +VICTIM: readahead +VICTIM: directio_read +VICTIM: directio_write +VICTIM: writeback +Remount and restart healer +Try buffered read again +Try directio read again +Try directio write again +Try buffered write again +Kill healer again
tests/xfs/1897 new file +98 -0
98 lines not on this page · raw message
tests/xfs/1897.out new file +4 -0
@@ -0,0 +1,4 @@ +QA output created by 1897 +Simulate media error +Kill healer +SCRATCH_MNT: media error on DEVICE daddr NUM bbcount NUM
tests/xfs/1898 new file +36 -0
@@ -0,0 +1,36 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2024-2025 Oracle. All Rights Reserved. +# +# FS QA Test No. 1898 +# +# Check that xfs_healer can report filesystem shutdowns. + +. ./common/preamble +_begin_fstest auto quick scrub eio selfhealing + +. ./common/fuzzy +. ./common/filter +. ./common/systemd + +_require_scratch_nocheck +_require_scrub +_require_command "$XFS_HEALER_PROG" "xfs_healer" + +_scratch_mkfs >> $seqres.full +_scratch_mount +$XFS_IO_PROG -f -c "pwrite -S 0x58 0 500k" -c "fsync" $victim >> $seqres.full + +echo "Start healer and shut down" +_scratch_invoke_xfs_healer "$tmp.healer" +_scratch_shutdown -f + +# Unmount filesystem to start fresh +echo "Kill healer" +_scratch_kill_xfs_healer +cat $tmp.healer >> $seqres.full +cat $tmp.healer | _filter_scratch | grep 'shut down' + +# success, all done +status=0 +exit
tests/xfs/1898.out new file +4 -0
@@ -0,0 +1,4 @@ +QA output created by 1898 +Start healer and shut down +Kill healer +SCRATCH_MNT: filesystem shut down due to forced unmount
tests/xfs/1899 new file +109 -0
109 lines not on this page · raw message
tests/xfs/1899.out new file +3 -0
@@ -0,0 +1,3 @@ +QA output created by 1899 +ls: reading directory 'SCRATCH_MNT/some/victimdir': Structure needs cleaning +SCRATCH_MNT: Full repair: Repairs in progress.
tests/xfs/1900 new file +116 -0
116 lines not on this page · raw message
tests/xfs/1900.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1900 +ls: reading directory 'TEST_DIR/moocow/some/victimdir': Structure needs cleaning
tests/xfs/1901 new file +138 -0
138 lines not on this page · raw message
tests/xfs/1901.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1901 +ls: reading directory 'MNTPT/some/victimdir': Structure needs cleaning