xfs: autonomous self healing
Subsystems (6)
PATCH v2 1/1 xfs: autonomous self healing
Hi everyone,
You might recall that 18 months ago I showed off an early draft of a patchset implementing autonomous self healing capabilities for XFS. The premise is quite simple -- add a few hooks to the kernel to capture significant filesystem metadata and file health events (pretty much all failures), queue these events to a special anonfd, and let userspace read the events at its leisure. That's patchset 1.
Since the previous release, I've removed all the json event generation stuff and made media errors use the rmap btree to report file data loss. I also ported the userspace program to C. I'm not going to blast everyone with the full set; just know that the C version is here:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
Patchset 2 is now a cleanup of the file IO error hooks in patchset 1 to use a more generic interface and to call fsnotify with the error reports. This means that the fsnotify filesystem error functionality conveys generic errors to unprivileged userspace programs, but I'm leaving the privileged healthmon interface so that xfsprogs can figure out which specific part of the filesystem needs fixing.
This work was mostly complete by the end of 2024, and I've been letting it run on my XFS QA testing fleets ever since then. I am submitting this patchset for upstream for 6.19. Once this is merged, the online fsck project will be complete.
--D
The unreviewed patches in this series are:
[PATCHSET V3 1/2] xfs: autonomous self healing of filesystems
[PATCH 02/22] docs: discuss autonomous self healing in the xfs online [PATCH 03/22] xfs: create debugfs uuid aliases
[PATCH 04/22] xfs: create hooks for monitoring health updates [PATCH 05/22] xfs: create a filesystem shutdown hook [PATCH 06/22] xfs: create hooks for media errors
[PATCH 07/22] iomap: report buffered read and write io errors to the [PATCH 08/22] iomap: report directio read and write errors to callers [PATCH 09/22] xfs: create file io error hooks
[PATCH 10/22] xfs: create a special file to pass filesystem health to [PATCH 11/22] xfs: create event queuing, formatting, [PATCH 12/22] xfs: report metadata health events through healthmon [PATCH 13/22] xfs: report shutdown events through healthmon [PATCH 14/22] xfs: report media errors through healthmon [PATCH 15/22] xfs: report file io errors through healthmon [PATCH 16/22] xfs: allow reconfiguration of the health monitoring [PATCH 17/22] xfs: validate fds against running healthmon [PATCH 18/22] xfs: add media error reporting ioctl
[PATCH 19/22] xfs: send uevents when major filesystem events happen [PATCH 20/22] xfs: merge health monitoring events when possible [PATCH 21/22] xfs: restrict healthmon users further
[PATCH 22/22] xfs: charge healthmon event objects to the memcg of the
[PATCHSET V3 2/2] iomap: generic file IO error reporting
[PATCH 1/6] iomap: report file IO errors to fsnotify [PATCH 2/6] xfs: switch healthmon to use the iomap I/O error [PATCH 3/6] xfs: port notify-failure to use the new vfs io error [PATCH 4/6] xfs: remove file I/O error hooks
[PATCH 5/6] iomap: remove I/O error hooks
[PATCH 6/6] xfs: report fs metadata errors via fsnotify
PATCH v3 1/2 xfs: autonomous self healing of filesystems
Hi all,
This patchset builds new functionality to deliver live information about filesystem health events to userspace. This is done by creating an anonymous file that can be read() for events by userspace programs. Events are captured by hooking various parts of XFS and iomap so that metadata health failures, file I/O errors, and major changes in filesystem state (unmounts, shutdowns, etc.) can be observed by programs.
When an event occurs, the hook functions queue an event object to each event anonfd for later processing. Programs must have CAP_SYS_ADMIN to open the anonfd and there's a maximum event lag to prevent resource overconsumption. The events themselves can be read() from the anonfd either as json objects for human readability, or as C structs for daemons.
In userspace, we create a new daemon program that will read the event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some udev rules.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* docs: remove obsolete links in the xfs online repair documentation
* docs: discuss autonomous self healing in the xfs online repair design doc
* xfs: create debugfs uuid aliases
* xfs: create hooks for monitoring health updates
* xfs: create a filesystem shutdown hook
* xfs: create hooks for media errors
* iomap: report buffered read and write io errors to the filesystem
* iomap: report directio read and write errors to callers
* xfs: create file io error hooks
* xfs: create a special file to pass filesystem health to userspace
* xfs: create event queuing, formatting, and discovery infrastructure
* xfs: report metadata health events through healthmon
* xfs: report shutdown events through healthmon
* xfs: report media errors through healthmon
* xfs: report file io errors through healthmon
* xfs: allow reconfiguration of the health monitoring device
* xfs: validate fds against running healthmon
* xfs: add media error reporting ioctl
* xfs: send uevents when major filesystem events happen
* xfs: merge health monitoring events when possible
* xfs: restrict healthmon users further
* xfs: charge healthmon event objects to the memcg of the listening process
--- fs/iomap/internal.h | 2 fs/xfs/libxfs/xfs_fs.h | 171 ++ fs/xfs/libxfs/xfs_health.h | 52 + fs/xfs/xfs_file.h | 37 + fs/xfs/xfs_fsops.h | 11 fs/xfs/xfs_healthmon.h | 108 + fs/xfs/xfs_linux.h | 3 fs/xfs/xfs_mount.h | 13 fs/xfs/xfs_notify_failure.h | 39 + fs/xfs/xfs_super.h | 13 fs/xfs/xfs_trace.h | 407 ++++++ include/linux/fs.h | 4 include/linux/iomap.h | 2 Documentation/filesystems/vfs.rst | 7 .../filesystems/xfs/xfs-online-fsck-design.rst | 450 +++--- fs/iomap/buffered-io.c | 27 fs/iomap/direct-io.c | 4 fs/iomap/ioend.c | 4 fs/xfs/Kconfig | 8 fs/xfs/Makefile | 7 fs/xfs/xfs_aops.c | 2 fs/xfs/xfs_file.c | 174 ++ fs/xfs/xfs_fsops.c | 60 + fs/xfs/xfs_health.c | 271 ++++ fs/xfs/xfs_healthmon.c | 1466 ++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 7 fs/xfs/xfs_notify_failure.c | 258 +++- fs/xfs/xfs_super.c | 109 + fs/xfs/xfs_trace.c | 4 lib/seq_buf.c | 1 30 files changed, 3477 insertions(+), 244 deletions(-) create mode 100644 fs/xfs/xfs_healthmon.h create mode 100644 fs/xfs/xfs_healthmon.c
PATCH 1/22 docs: remove obsolete links in the xfs online repair documentation
From: Darrick J. Wong <[email protected]>
Online repair is now merged in upstream, no need to point to patchset links anymore.
--- .../filesystems/xfs/xfs-online-fsck-design.rst | 236 +------------------- 1 file changed, 6 insertions(+), 230 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +6 -230
@@ -105,10 +105,8 @@ occur; this capability aids both strategies. TLDR; Show Me the Code! ----------------------- -Code is posted to the kernel.org git trees as follows: -`kernel changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink>`_, -`userspace changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_, and -`QA test changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs>`_. +Kernel and userspace code has been fully merged as of October 2025. + Each kernel patchset adding an online repair function will use the same branch name across the kernel, xfsprogs, and fstests git repos.
@@ -764,12 +762,8 @@ allow the online fsck developers to compare online fsck against offline fsck, and they enable XFS developers to find deficiencies in the code base. Proposed patchsets include -`general fuzzer improvements -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_, `fuzzing baselines -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_, -and `improvements in fuzz testing comprehensiveness -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_. +<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_. Stress Testing --------------
@@ -801,11 +795,6 @@ Success is defined by the ability to run all of these tests without observing any unexpected filesystem shutdowns due to corrupted metadata, kernel hang check warnings, or any other sort of mischief. -Proposed patchsets include `general stress testing -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_ -and the `evolution of existing per-function stress testing -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_. - 4. User Interface =================
@@ -886,10 +875,6 @@ apply as nice of a priority to IO and CPU scheduling as possible. This measure was taken to minimize delays in the rest of the filesystem. No such hardening has been performed for the cron job. -Proposed patchset: -`Enabling the xfs_scrub background service -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_. - Health Reporting ----------------
@@ -912,13 +897,6 @@ notifications and initiate a repair? *Answer*: These questions remain unanswered, but should be a part of the conversation with early adopters and potential downstream users of XFS. -Proposed patchsets include -`wiring up health reports to correction returns -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_ -and -`preservation of sickness info during memory reclaim -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_. - 5. Kernel Algorithms and Data Structures ========================================
@@ -1310,21 +1288,6 @@ Space allocation records are cross-referenced as follows: are there the same number of reverse mapping records for each block as the reference count record claims? -Proposed patchsets are the series to find gaps in -`refcount btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_, -`inode btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and -`rmap btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records; -to find -`mergeable records -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_; -and to -`improve cross referencing with rmap -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_ -before starting a repair. - Checking Extended Attributes ````````````````````````````
@@ -1756,10 +1719,6 @@ For scrub, the drain works as follows: To avoid polling in step 4, the drain provides a waitqueue for scrub threads to be woken up whenever the intent count drops to zero. -The proposed patchset is the -`scrub intent drain series -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_. - .. _jump_labels: Static Keys (aka Jump Label Patching)
@@ -2036,10 +1995,6 @@ The ``xfarray_store_anywhere`` function is used to insert a record in any null record slot in the bag; and the ``xfarray_unset`` function removes a record from the bag. -The proposed patchset is the -`big in-memory array -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_. - Iterating Array Elements ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2172,10 +2127,6 @@ However, it should be noted that these repair functions only use blob storage to cache a small number of entries before adding them to a temporary ondisk file, which is why compaction is not required. -The proposed patchset is at the start of the -`extended attribute repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series. - .. _xfbtree: In-Memory B+Trees
@@ -2214,11 +2165,6 @@ xfiles enables reuse of the entire btree library. Btrees built atop an xfile are collectively known as ``xfbtrees``. The next few sections describe how they actually work. -The proposed patchset is the -`in-memory btree -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_ -series. - Using xfiles as a Buffer Cache Target ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2459,14 +2405,6 @@ This enables the log to release the old EFI to keep the log moving forwards. EFIs have a role to play during the commit and reaping phases; please see the next section and the section about :ref:`reaping<reaping>` for more details. -Proposed patchsets are the -`bitmap rework -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_ -and the -`preparation for bulk loading btrees -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_. - - Writing the New Tree ````````````````````
@@ -2623,11 +2561,6 @@ The number of records for the inode btree is the number of xfarray records, but the record count for the free inode btree has to be computed as inode chunk records are stored in the xfarray. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding the Space Reference Counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2716,11 +2649,6 @@ Reverse mappings are added to the bag using ``xfarray_store_anywhere`` and removed via ``xfarray_unset``. Bag members are examined through ``xfarray_iter`` loops. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding File Fork Mapping Indices ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -2757,11 +2685,6 @@ EXTENTS format instead of BMBT, which may require a conversion. Third, the incore extent map must be reloaded carefully to avoid disturbing any delayed allocation extents. -The proposed patchset is the -`file mapping repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_ -series. - .. _reaping: Reaping Old Metadata Blocks
@@ -2843,11 +2766,6 @@ blocks. As stated earlier, online repair functions use very large transactions to minimize the chances of this occurring. -The proposed patchset is the -`preparation for bulk loading btrees -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_ -series. - Case Study: Reaping After a Regular Btree Repair ````````````````````````````````````````````````
@@ -2943,11 +2861,6 @@ When the walk is complete, the bitmap disunion operation ``(ag_owner_bitmap & btrees. These blocks can then be reaped using the methods outlined above. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - .. _rmap_reap: Case Study: Reaping After Repairing Reverse Mapping Btrees
@@ -2972,11 +2885,6 @@ methods outlined above. The rest of the process of rebuildng the reverse mapping btree is discussed in a separate :ref:`case study<rmap_repair>`. -The proposed patchset is the -`AG btree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_ -series. - Case Study: Rebuilding the AGFL ```````````````````````````````
@@ -3024,11 +2932,6 @@ more complicated, because computing the correct value requires traversing the forks, or if that fails, leaving the fields invalid and waiting for the fork fsck functions to run. -The proposed patchset is the -`inode -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_ -repair series. - Quota Record Repairs --------------------
@@ -3045,11 +2948,6 @@ checking are obviously bad limits and timer values. Quota usage counters are checked, repaired, and discussed separately in the section about :ref:`live quotacheck <quotacheck>`. -The proposed patchset is the -`quota -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_ -repair series. - .. _fscounters: Freezing to Fix Summary Counters
@@ -3145,11 +3043,6 @@ long enough to check and correct the summary counters. | This bug was fixed in Linux 5.17. | +--------------------------------------------------------------------------+ -The proposed patchset is the -`summary counter cleanup -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_ -series. - Full Filesystem Scans ---------------------
@@ -3277,15 +3170,6 @@ Second, if the incore inode is stuck in some intermediate state, the scan coordinator must release the AGI and push the main filesystem to get the inode back into a loadable state. -The proposed patches are the -`inode scanner -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ -series. -The first user of the new functionality is the -`online quotacheck -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ -series. - Inode Management ````````````````
@@ -3381,12 +3265,6 @@ To capture these nuances, the online fsck code has a separate ``xchk_irele`` function to set or clear the ``DONTCACHE`` flag to get the required release behavior. -Proposed patchsets include fixing -`scrub iget usage -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_ and -`dir iget usage -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_. - .. _ilocking: Locking Inodes
@@ -3443,11 +3321,6 @@ If the dotdot entry changes while the directory is unlocked, then a move or rename operation must have changed the child's parentage, and the scan can exit early. -The proposed patchset is the -`directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ -series. - .. _fshooks: Filesystem Hooks
@@ -3594,11 +3467,6 @@ The inode scan APIs are pretty simple: - ``xchk_iscan_teardown`` to finish the scan -This functionality is also a part of the -`inode scanner -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_ -series. - .. _quotacheck: Case Study: Quota Counter Checking
@@ -3686,11 +3554,6 @@ needing to hold any locks for a long duration. If repairs are desired, the real and shadow dquots are locked and their resource counts are set to the values in the shadow dquot. -The proposed patchset is the -`online quotacheck -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_ -series. - .. _nlinks: Case Study: File Link Count Checking
@@ -3744,11 +3607,6 @@ shadow information. If no parents are found, the file must be :ref:`reparented <orphanage>` to the orphanage to prevent the file from being lost forever. -The proposed patchset is the -`file link count repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_ -series. - .. _rmap_repair: Case Study: Rebuilding Reverse Mapping Records
@@ -3828,11 +3686,6 @@ scan for reverse mapping records. 12. Free the xfbtree now that it not needed. -The proposed patchset is the -`rmap repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_ -series. - Staging Repairs with Temporary Files on Disk --------------------------------------------
@@ -3971,11 +3824,6 @@ Once a good copy of a data file has been constructed in a temporary file, it must be conveyed to the file being repaired, which is the topic of the next section. -The proposed patches are in the -`repair temporary files -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_ -series. - Logged File Content Exchanges -----------------------------
@@ -4025,11 +3873,6 @@ The new ``XFS_SB_FEAT_INCOMPAT_EXCHRANGE`` incompatible feature flag in the superblock protects these new log item records from being replayed on old kernels. -The proposed patchset is the -`file contents exchange -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_ -series. - +--------------------------------------------------------------------------+ | **Sidebar: Using Log-Incompatible Feature Flags** | +--------------------------------------------------------------------------+
@@ -4323,11 +4166,6 @@ To repair the summary file, write the xfile contents into the temporary file and use atomic mapping exchange to commit the new contents. The temporary file is then reaped. -The proposed patchset is the -`realtime summary repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_ -series. - Case Study: Salvaging Extended Attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4369,11 +4207,6 @@ Salvaging extended attributes is done as follows: 4. Reap the temporary file. -The proposed patchset is the -`extended attribute repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ -series. - Fixing Directories ------------------
@@ -4448,11 +4281,6 @@ Unfortunately, the current dentry cache design doesn't provide a means to walk every child dentry of a specific directory, which makes this a hard problem. There is no known solution. -The proposed patchset is the -`directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_ -series. - Parent Pointers ```````````````
@@ -4612,11 +4440,6 @@ a :ref:`directory entry live update hook <liveupdate>` as follows: 7. Reap the temporary directory. -The proposed patchset is the -`parent pointers directory repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ -series. - Case Study: Repairing Parent Pointers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4662,11 +4485,6 @@ directory reconstruction: 8. Reap the temporary file. -The proposed patchset is the -`parent pointers repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_ -series. - Digression: Offline Checking of Parent Pointers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4755,11 +4573,6 @@ connectivity checks: 4. Move on to examining link counts, as we do today. -The proposed patchset is the -`offline parent pointers repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_ -series. - Rebuilding directories from parent pointers in offline repair would be very challenging because xfs_repair currently uses two single-pass scans of the filesystem during phases 3 and 4 to decide which files are corrupt enough to be
@@ -4903,12 +4716,6 @@ Repairing the directory tree works as follows: 6. If the subdirectory has zero paths, attach it to the lost and found. -The proposed patches are in the -`directory tree repair -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_ -series. - - .. _orphanage: The Orphanage
@@ -4973,11 +4780,6 @@ Orphaned files are adopted by the orphanage as follows: 7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all resources. -The proposed patches are in the -`orphanage adoption -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_ -series. - 6. Userspace Algorithms and Data Structures ===========================================
@@ -5091,14 +4893,6 @@ first workqueue's workers until the backlog eases. This doesn't completely solve the balancing problem, but reduces it enough to move on to more pressing issues. -The proposed patchsets are the scrub -`performance tweaks -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_ -and the -`inode scan rebalance -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_ -series. - .. _scrubrepair: Scheduling Repairs
@@ -5179,20 +4973,6 @@ immediately. Corrupt file data blocks reported by phase 6 cannot be recovered by the filesystem. -The proposed patchsets are the -`repair warning improvements -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_, -refactoring of the -`repair data dependency -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_ -and -`object tracking -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_, -and the -`repair scheduling -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_ -improvement series. - Checking Names for Confusable Unicode Sequences -----------------------------------------------
@@ -5372,6 +5152,8 @@ The extra flexibility enables several new use cases: This emulates an atomic device write in software, and can support arbitrary scattered writes. +(This functionality was merged into mainline as of 2025) + Vectorized Scrub ----------------
@@ -5393,13 +5175,7 @@ It is hoped that ``io_uring`` will pick up enough of this functionality that online fsck can use that instead of adding a separate vectored scrub system call to XFS. -The relevant patchsets are the -`kernel vectorized scrub -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_ -and -`userspace vectorized scrub -<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_ -series. +(This functionality was merged into mainline as of 2025) Quality of Service Targets for Scrub ------------------------------------
Re: [PATCH 01/22] docs: remove obsolete links in the xfs online repair documentation
Quoted 7 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Online repair is now merged in upstream, no need to point to patchset links anymore.
Signed-off-by "Darrick J. Wong" <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>
Can we get this queued up for 6.19? Sending it standalone might be best, although given that Carlos is on Cc maybe he can directly pick it up?
Carlos Maiolino
maintainerRe: [PATCH 01/22] docs: remove obsolete links in the xfs online repair documentation
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Online repair is now merged in upstream, no need to point to patchset links anymore.
Quoted 7 lines
Quoted 2 lines
Signed-off-by "Darrick J. Wong" <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Can we get this queued up for 6.19? Sending it standalone might be best, although given that Carlos is on Cc maybe he can directly pick it up?
Done, will send an update soon.
PATCH 2/22 docs: discuss autonomous self healing in the xfs online repair design doc
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Update the XFS online repair document to describe the motivation and design of the autonomous filesystem healing agent known as xfs_healer.
--- .../filesystems/xfs/xfs-online-fsck-design.rst | 218 ++++++++++++++++++++ 1 file changed, 216 insertions(+), 2 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +216 -2
@@ -166,9 +166,12 @@ The current XFS tools leave several problems unsolved: malicious actors **exploit quirks of Unicode** to place misleading names in directories. +8. **Site Reliability and Support Engineers** would like to reduce the + frequency of incidents requiring **manual intervention**. + Given this definition of the problems to be solved and the actors who would benefit, the proposed solution is a third fsck tool that acts on a running -filesystem. +filesystem, and an autononmous agent that fixes problems as they arise. This new third program has three components: an in-kernel facility to check metadata, an in-kernel facility to repair metadata, and a userspace driver
@@ -203,6 +206,13 @@ Even if a piece of filesystem metadata can only be regenerated by scanning the entire system, the scan can still be done in the background while other file operations continue. +The autonomous self healing agent should listen for metadata health impact +reports coming from the kernel and automatically schedule repairs for the +damaged metadata. +If the required repairs are larger in scope than a single metadata structure, +``xfs_scrub`` should be invoked to perform a full analysis. +``xfs_healer`` is the name of this program. + In summary, online fsck takes advantage of resource sharding and redundant metadata to enable targeted checking and repair operations while the system is running.
@@ -850,11 +860,16 @@ variable in the following service files: * ``xfs_scrub_all_fail.service`` The decision to enable the background scan is left to the system administrator. -This can be done by enabling either of the following services: +This can be done system-wide by enabling either of the following services: * ``xfs_scrub_all.timer`` on systemd systems * ``xfs_scrub_all.cron`` on non-systemd systems +To enable online repair for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To enable only scanning, the property should be set to ``check``. +To disable online fsck entirely, the property should be set to ``none``. + This automatic weekly scan is configured out of the box to perform an additional media scan of all file data once per month. This is less foolproof than, say, storing file data block checksums, but much
@@ -897,6 +912,36 @@ notifications and initiate a repair? *Answer*: These questions remain unanswered, but should be a part of the conversation with early adopters and potential downstream users of XFS. +Autonomous Self Healing +----------------------- + +The autonomous self healing agent is a background system service that starts +when the filesystem is mounted and runs until unmount. +When starting up, the agent opens a special pseudofile under the specific +mount. +When the filesystem generates new adverse health events, the events will be +made available for reading via the special pseudofile. +The events need not be limited to metadata concerns; they can also reflect +events outside of the filesystem's direct control such as file I/O errors. + +The agent reads these events in a loop and responds to the events +appropriately. +For a single trouble report about metadata, the agent initiates a targeted +repair of the specific structure. +If that repair fails or the agent observes too many metadata trouble reports +over a short interval, it should then initiate a full scan of the filesystem +via the ``xfs_scrub`` service. + +The decision to enable the background scan is left to the system administrator. +This can be done system-wide by enabling the following services: + +* ``[email protected]`` on systemd systems + +To enable autonomous healing for specific filesystems, the ``autofsck`` +filesystem property should be set to ``repair``. +To disable self healing, the property should be set to ``check``, +``optimize``, or ``none``. + 5. Kernel Algorithms and Data Structures ========================================
@@ -4780,6 +4825,136 @@ Orphaned files are adopted by the orphanage as follows: 7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all resources. +Health Monitoring +----------------- + +A self-correcting filesystem responds to observations of problems by scheduling +repairs of the affected areas. +The filesystem must therefore create event objects in response to stimuli +(metadata corruption, file I/O errors, etc.) and dispatch these events to +downstream consumers. +Downstream consumers that are in the kernel itself are easy to implement with +the ``xfs_hooks`` infrastructure created for other parts of online repair; these +are basically indirect function calls. + +However, the decision to translate an adverse metadata health report into a +repair should be made by userspace, and the actual scheduling done by userspace. +Some users (e.g. containers) would prefer to fast-fail the container and restart +it on another node at a previous checkpoint. +For workloads running in isolation, repairs may be preferable; either way this +is something the system administrator knows, and not the kernel. +A userspace agent (``xfs_healer``, described later) will collect events from the +kernel and dispatch them appropriately. + +Exporting health events to userspace requires the creation of a new component, +known as the health monitor. +Because the monitor exposes itself to userspace to deliver information, a file +descriptor is the natural abstraction to use here. +The health monitor hooks all the relevant sources of metadata health events. +Upon activation of the hook, a new event object is created and added to a queue. +When the agent reads from the fd, event objects are pulled from the start of the +queue and formatted into the user's buffer. +The events are freed, and the read call returns to userspace to allow the agent +to perform some work. +Memory usage is constrained on a per-fd basis to prevent memory exhaustion; if +an event must be discarded, a special "lost event" event is delivered to the +agent. + +In short, health events are captured, queued, and eventually copied out to +userspace for dispatching. + +**Question**: Why use a pseudofile and not use existing notification methods? + +*Answer*: The pseudofile is a private filesystem interface only available to +processes with the CAP_SYS_ADMIN priviledge and the ability to open the root +directory. +Being private gives the kernel and ``xfs_healer`` the flexibility to change +or update the event format in the future without worrying about backwards +compatibility. +Using existing notifications means that the event format would be frozen in +the public fsnotify UAPI forever, which would affect two subsystems. + +The pseudofile can also accept ioctls, which gives ``xfs_healer`` a solid +means to validate that prior to a repair, its reopened mountpoint is actually +the same filesystem that is being monitored. + +**Question**: Why not reuse fs/notify? + +*Answer*: It's much simpler for the healthmon code to manage its own queue of +events and to wake up readers instead of reusing fsnotify because that's the +only part of fsnotify that would use. + +Before I get started, an introduction: fsnotify expects its users (e.g. +fanotify) to implement quite a bit of functionality; all it provides is a +wrapper around a simple queue and a lot of code to convey information about the +calling process to that user. +fanotify has to actually implement all the queue management code on its own, +and so would healthmon. + +So if healthmon used fsnotify, it would have to create its own fsnotify group +structure. +For our purposes, the group is a very large wrapper around a linked list, some +counters, and a mutex. +The group object is critical for ensuring that sees only its own events, and +that nobody else (e.g. regular fanotify) ever sees these events. +There's a lot more in there for controlling whether fanotify reports pids, +groups, file handles, etc. that healthmon doesn't care about. + +Starting from the fsnotify() function call: + + - I /think/ we'd have to define a new "data type", which itself is just a plain + int but I think they correspond to FSNOTIFY_EVENT_* values which themselves + are actually part of an enum. + The data type controls the typecasting options for the ``void *data`` + parameter, which I guess is how I'd pass the healthmon event info from the + hooks into the fsnotify mechanism and back out to the healthmon code. + + - Each filesystem that wants to do this probably has to add their own + FSNOTIFY_EVENT_{XFS,BTRFS,BFS} data type value because that's a casting + decision that's made inside the main fsnotify code. + I think this can be avoided if each fs is careful never to leak events + outside of the group. + Either way, it's harder to follow the data flows here because fsnotify can + only take and pass around ``void *`` pointers, and it makes various indirect + function calls to manage events. + Contrast this with doing everything with typed pointers and direct calls + within ``xfs_healthmon.c``. + + - Since healthmon is both producer and consumer of fsnotify events, we can + probably define our own "mask" value. + It's a relief that we don't have to interact with fanotify, because fanotify + has used up 22 of its 32 mask bits. + +Once healthmon gets an event into fsnotify, fsnotify will call back (into +healthmon!) to tell it that it got an event. +From there, the fsnotify implementation (healthmon) has to allocate an event +object and add it to the event queue in the group, which is what it already does +now. +Overflow control is up to the fsnotify implementation, which healthmon already +implements. + +After the event is queued, the fsnotify implementation also has to implement its +own read file op to dequeue an event and copy it to the userspace buffer in +whatever format it likes. +Again, healthmon already does all this. + +In the end, replacing the homegrown event dispatching in healthmon with fsnotify +would make the data flows much harder to understand, and all we gain is a +generic event dispatcher that relies on indirect function calls instead of +direct ones. +We still have to implement the queuing discipline ourselves! :( + +**Future Work Question**: Should these events be exposed through the fanotify +filesystem error event interface? + +*Answer*: Yes. +fanotify is much more careful about filtering out events to processes that +aren't running with privileges. +These processes should have a means to receive simple notifications about +file errors. +However, this will require coordination between fanotify, ext4, and XFS, and +is (for now) outside the scope of this project. + 6. Userspace Algorithms and Data Structures ===========================================
@@ -5071,6 +5246,45 @@ and report what has been lost. For media errors in blocks owned by files, parent pointers can be used to construct file paths from inode numbers for user-friendly reporting. +Autonomous Self Healing +----------------------- + +When a filesystem mounts, the Linux kernel initiates a uevent describing the +mount and the path to the data device. +A udev rule determines the initial mountpoint from the data device path +and starts a mount-specific ``xfs_healer`` service instance. +The ``xfs_healer`` service opens the mountpoint and issues the +XFS_IOC_HEALTH_MONITOR ioctl to open a special health monitoring file. +After that is set up, the mountpoint is closed to avoid pinning the mount. + +The health monitoring file hooks certain points of the filesystem so that it +may receive events about metadata health, filesystem shutdowns, media errors, +file I/O errors, and unmounting of the filesystem. +Events are queued up for each health monitor file and encoded into a +``struct xfs_health_monitor_event`` object when the agent calls ``read()`` on +the file. +All health events are dispatched to a background threadpool to reduce stalls +in the main event loop. +Events can be logged into the system log for further analysis. + +For metadata health events, the specific details are used to construct a call +to the scrub ioctl. +The filesystem mountpoint is reopened, and the kernel is called. +If events are lost or the repairs fail, a full scan will be initiated by +starting up an ``[email protected]`` for the given mountpoint. + +A filesystem shutdown causes all future repair work to cease, and an unmount +causes the agent to exit. + +**Future Work Question**: Should the healer daemon also register a dbus +listener and publish events there? + +*Answer*: This is unclear -- if there's a demand for system monitoring daemons +to consume this information and make decisions, then yes, this could be wired +up in ``xfs_healer``. +On the other hand, systemd is in the middle of a transition to varlink, so +it makes more sense to wait and see what happens. + 7. Conclusion and Future Work =============================
PATCH 3/22 xfs: create debugfs uuid aliases
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an alias for the debugfs dir so that we can find a filesystem by uuid. Unless it's mounted nouuid.
--- fs/xfs/xfs_mount.h | 1 + fs/xfs/xfs_super.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
fs/xfs/xfs_mount.h +1 -0
@@ -289,6 +289,7 @@ typedef struct xfs_mount {struct delayed_work m_reclaim_work;/* background inode reclaim */ struct xfs_zone_info *m_zone_info;/* zone allocator information */ struct dentry *m_debugfs;/* debugfs parent */ +struct dentry *m_debugfs_uuid;/* debugfs symlink */ struct xfs_kobj m_kobj;struct xfs_kobj m_error_kobj;struct xfs_kobj m_error_meta_kobj;
fs/xfs/xfs_super.c +11 -0
@@ -819,6 +819,7 @@ xfs_mount_free(if (mp->m_ddev_targp)xfs_free_buftarg (mp->m_ddev_targp); +debugfs_remove (mp->m_debugfs_uuid);debugfs_remove (mp->m_debugfs);kfree (mp->m_rtname);kfree (mp->m_logname);
@@ -1969,6 +1970,16 @@ xfs_fs_fill_super( goto out_unmount; } +if (xfs_debugfs && mp->m_debugfs && !xfs_has_nouuid (mp)) { +char name[UUID_STRING_LEN +1 ]; + +snprintf (name,UUID_STRING_LEN +1 ,"%pU" , &mp->m_sb.sb_uuid); + mp->m_debugfs_uuid =debugfs_create_symlink (name, xfs_debugfs, + mp->m_super->s_id); + }else { + mp->m_debugfs_uuid =NULL ; + } +return 0 ; out_filestream_unmount:
PATCH 4/22 xfs: create hooks for monitoring health updates
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create hooks for monitoring health events.
--- fs/xfs/libxfs/xfs_health.h | 47 ++++++++++ fs/xfs/xfs_mount.h | 3 + fs/xfs/xfs_health.c | 204 ++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_super.c | 1 4 files changed, 254 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_health.h +47 -0
@@ -289,4 +289,51 @@ void xfs_bulkstat_health(struct xfs_inode *ip, struct xfs_bulkstat *bs);#define xfs_metadata_is_sick (error) \ (unlikely((error) == -EFSCORRUPTED || (error) == -EFSBADCRC)) +/* +* Parameters for tracking health updates. The enum below is passed as the +* hook function argument. +*/ +enum xfs_health_update_type { +XFS_HEALTHUP_SICK =1 ,/* runtime corruption observed */ +XFS_HEALTHUP_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHUP_HEALTHY ,/* fsck reported healthy structure */ +XFS_HEALTHUP_UNMOUNT ,/* filesystem is unmounting */ +}; + +/* Where in the filesystem was the event observed? */ +enum xfs_health_update_domain { +XFS_HEALTHUP_FS =1 ,/* main filesystem */ +XFS_HEALTHUP_AG ,/* allocation group */ +XFS_HEALTHUP_INODE ,/* inode */ +XFS_HEALTHUP_RTGROUP ,/* realtime group */ +}; + +struct xfs_health_update_params { +/* XFS_HEALTHUP_INODE */ +xfs_ino_t ino; +uint32_t gen; + +/* XFS_HEALTHUP_AG/RTGROUP */ +uint32_t group; + +/* XFS_SICK_* flags */ +unsigned int old_mask; +unsigned int new_mask; + +enum xfs_health_update_domain domain; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_health_hook { +struct xfs_hook health_hook; +}; + +void xfs_health_hook_disable (void ); +void xfs_health_hook_enable (void ); + +int xfs_health_hook_add (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_del (struct xfs_mount *mp,struct xfs_health_hook *hook); +void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_HEALTH_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -343,6 +343,9 @@ typedef struct xfs_mount {/* Hook to feed dirent updates to an active online repair. */ struct xfs_hooks m_dir_update_hooks; + +/* Hook to feed health events to a daemon. */ +struct xfs_hooks m_health_update_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_health.c +203 -1
350 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2291,6 +2291,7 @@ xfs_init_fs_context( mp->m_allocsize_log =16 ;/* 64k */ xfs_hooks_init (&mp->m_dir_update_hooks); +xfs_hooks_init (&mp->m_health_update_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 5/22 xfs: create a filesystem shutdown hook
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a hook so that health monitoring can report filesystem shutdown events to userspace. Shutdowns should be infrequent, so we don't bother with a static key here.
--- fs/xfs/xfs_fsops.h | 11 +++++++++++ fs/xfs/xfs_mount.h | 3 +++ fs/xfs/xfs_fsops.c | 42 ++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_super.c | 1 + 4 files changed, 57 insertions(+)
fs/xfs/xfs_fsops.h +11 -0
@@ -15,4 +15,15 @@ int xfs_fs_goingdown(struct xfs_mount *mp, uint32_t inflags);int xfs_fs_reserve_ag_blocks (struct xfs_mount *mp);void xfs_fs_unreserve_ag_blocks (struct xfs_mount *mp); +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_shutdown_hook { +struct xfs_hook shutdown_hook; +}; + +int xfs_shutdown_hook_add (struct xfs_mount *mp,struct xfs_shutdown_hook *hook); +void xfs_shutdown_hook_del (struct xfs_mount *mp,struct xfs_shutdown_hook *hook); +void xfs_shutdown_hook_setup (struct xfs_shutdown_hook *hook, +notifier_fn_t mod_fn); +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_FSOPS_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -346,6 +346,9 @@ typedef struct xfs_mount {/* Hook to feed health events to a daemon. */ struct xfs_hooks m_health_update_hooks; + +/* Hook to feed shutdown events to a daemon. */ +struct xfs_hooks m_shutdown_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_fsops.c +42 -0
54 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2291,6 +2291,7 @@ xfs_init_fs_context( mp->m_allocsize_log =16 ;/* 64k */ xfs_hooks_init (&mp->m_dir_update_hooks); +xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks); fc->s_fs_info = mp;
PATCH 6/22 xfs: create hooks for media errors
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a media error event hook so that we can send events to userspace. Media errors are not expected to be frequent, so we don't have a static key guarding them here either.
--- fs/xfs/xfs_mount.h | 3 ++ fs/xfs/xfs_notify_failure.h | 33 +++++++++++++++++++++ fs/xfs/xfs_notify_failure.c | 68 ++++++++++++++++++++++++++++++++++++++++--- fs/xfs/xfs_super.c | 1 + 4 files changed, 100 insertions(+), 5 deletions(-)
fs/xfs/xfs_mount.h +3 -0
@@ -349,6 +349,9 @@ typedef struct xfs_mount {/* Hook to feed shutdown events to a daemon. */ struct xfs_hooks m_shutdown_hooks; + +/* Hook to feed media error events to a daemon. */ +struct xfs_hooks m_media_error_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_notify_failure.h +33 -0
@@ -8,4 +8,37 @@extern const struct dax_holder_operations xfs_dax_holder_operations; +enum xfs_failed_device { +XFS_FAILED_DATADEV , +XFS_FAILED_LOGDEV , +XFS_FAILED_RTDEV , +}; + +#if defined(CONFIG_XFS_LIVE_HOOKS ) && defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX ) +struct xfs_media_error_params { +struct xfs_mount *mp; +enum xfs_failed_device fdev; +xfs_daddr_t daddr; +uint64_t bbcount; +bool pre_remove; +}; + +struct xfs_media_error_hook { +struct xfs_hook error_hook; +}; + +int xfs_media_error_hook_add (struct xfs_mount *mp, +struct xfs_media_error_hook *hook); +void xfs_media_error_hook_del (struct xfs_mount *mp, +struct xfs_media_error_hook *hook); +void xfs_media_error_hook_setup (struct xfs_media_error_hook *hook, +notifier_fn_t mod_fn); +#else +struct xfs_media_error_params { }; +struct xfs_media_error_hook { }; +# define xfs_media_error_hook_add (...) (0) +# define xfs_media_error_hook_del (...) ((void)0) +# define xfs_media_error_hook_setup (...) ((void)0) +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_notify_failure.c +63 -5
91 lines not on this page · raw message
fs/xfs/xfs_super.c +1 -0
@@ -2293,6 +2293,7 @@ xfs_init_fs_context(xfs_hooks_init (&mp->m_dir_update_hooks);xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks); +xfs_hooks_init (&mp->m_media_error_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 7/22 iomap: report buffered read and write io errors to the filesystem
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Provide a callback so that iomap can report read and write IO errors to the caller filesystem. For now this is only wired up for iomap as a testbed for XFS.
--- fs/iomap/internal.h | 2 ++ include/linux/fs.h | 4 ++++ Documentation/filesystems/vfs.rst | 7 +++++++ fs/iomap/buffered-io.c | 27 +++++++++++++++++++++++++-- fs/iomap/ioend.c | 4 ++++ 5 files changed, 42 insertions(+), 2 deletions(-)
fs/iomap/internal.h +2 -0
@@ -5,5 +5,7 @@#define IOEND_BATCH_SIZE 4096u32 iomap_finish_ioend_direct (struct iomap_ioend *ioend); +void iomap_mapping_ioerror (struct address_space *mapping,int direction, +loff_t pos,u64 len,int error);#endif /* _IOMAP_INTERNAL_H */
include/linux/fs.h +4 -0
@@ -477,6 +477,10 @@ struct address_space_operations { sector_t *span);void (*swap_deactivate)(struct file *file);int (*swap_rw)(struct kiocb *iocb,struct iov_iter *iter); + +/* Callback for dealing with IO errors during readahead or writeback */ +void (*ioerror)(struct address_space *mapping,int direction, +loff_t pos,u64 len,int error); };extern const struct address_space_operations empty_aops;
Documentation/filesystems/vfs.rst +7 -0
@@ -822,6 +822,8 @@ cache in your filesystem. The following members are defined: int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); + void (*ioerror)(struct address_space *mapping, int direction, + loff_t pos, u64 len, int error); }; ``read_folio``
@@ -1032,6 +1034,11 @@ cache in your filesystem. The following members are defined: ``swap_rw`` Called to read or write swap pages when SWP_FS_OPS is set. +``ioerror`` + Called to deal with IO errors during readahead or writeback. + This may be called from interrupt context, and without any + locks necessarily being held. + The File Object ===============
fs/iomap/buffered-io.c +25 -2
@@ -288,6 +288,14 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter, pos >=i_size_read (iter->inode); } +inline void iomap_mapping_ioerror (struct address_space *mapping,int direction, +loff_t pos,u64 len,int error) +{ +if (mapping && mapping->a_ops->ioerror) + mapping->a_ops->ioerror (mapping, direction, pos, len, + error); +} +/** * iomap_read_inline_data - copy inline data into the page cache * @iter: iteration structure
@@ -310,8 +318,11 @@ static int iomap_read_inline_data(const struct iomap_iter *iter,if (folio_test_uptodate (folio))return 0 ; -if (WARN_ON_ONCE (size > iomap->length)) +if (WARN_ON_ONCE (size > iomap->length)) { +iomap_mapping_ioerror (folio->mapping,READ , iomap->offset, + size, -EIO );return -EIO ; + }if (offset >0 )ifs_alloc (iter->inode, folio, iter->flags);
@@ -339,6 +350,10 @@ static void iomap_finish_folio_read(struct folio *folio, size_t off,spin_unlock_irqrestore (&ifs->state_lock, flags); } +if (error) +iomap_mapping_ioerror (folio->mapping,READ , +folio_pos (folio) + off, len, error); +if (finished)folio_end_read (folio, uptodate); }
@@ -558,11 +573,15 @@ static int iomap_read_folio_range(const struct iomap_iter *iter,const struct iomap *srcmap =iomap_iter_srcmap (iter);struct bio_vec bvec;struct bio bio; +int ret;bio_init (&bio, srcmap->bdev, &bvec,1 ,REQ_OP_READ ); bio.bi_iter.bi_sector =iomap_sector (srcmap, pos);bio_add_folio_nofail (&bio, folio, len,offset_in_folio (folio, pos)); -return submit_bio_wait (&bio); + ret =submit_bio_wait (&bio); +if (ret) +iomap_mapping_ioerror (folio->mapping,READ , pos, len, ret); +return ret; }#else static int iomap_read_folio_range(const struct iomap_iter *iter,
@@ -1674,6 +1693,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)u64 pos =folio_pos (folio);u64 end_pos = pos +folio_size (folio);u64 end_aligned =0 ; +loff_t orig_pos = pos;bool wb_pending = false;int error =0 ;u32 rlen;
@@ -1724,6 +1744,9 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)if (wb_pending) wpc->nr_folios++; +if (error && pos > orig_pos) +iomap_mapping_ioerror (inode->i_mapping,WRITE , orig_pos,0 , + error);/* * We can have dirty bits set past end of file in page_mkwrite path
fs/iomap/ioend.c +4 -0
@@ -55,6 +55,10 @@ static u32 iomap_finish_ioend_buffered(struct iomap_ioend *ioend)/* walk all folios in bio, ending page IO on them */ bio_for_each_folio_all (fi, bio) { +if (ioend->io_error) +iomap_mapping_ioerror (inode->i_mapping,WRITE , +folio_pos (fi.folio) + fi.offset, + fi.length, ioend->io_error);iomap_finish_folio_write (inode, fi.folio, fi.length); folio_count++; }
PATCH 8/22 iomap: report directio read and write errors to callers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add more hooks to report directio IO errors to the filesystem.
--- include/linux/iomap.h | 2 ++ fs/iomap/direct-io.c | 4 ++++ 2 files changed, 6 insertions(+)
include/linux/iomap.h +2 -0
@@ -486,6 +486,8 @@ struct iomap_dio_ops { unsigned flags);void (*submit_io)(const struct iomap_iter *iter,struct bio *bio,loff_t file_offset); +void (*ioerror)(struct inode *inode,int direction,loff_t pos, +u64 len,int error);/* * Filesystems wishing to attach private information to a direct io bio
fs/iomap/direct-io.c +4 -0
@@ -95,6 +95,10 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)if (dops && dops->end_io) ret = dops->end_io (iocb, dio->size, ret, dio->flags); +if (dio->error && dops && dops->ioerror) + dops->ioerror (file_inode (iocb->ki_filp), + (dio->flags &IOMAP_DIO_WRITE ) ?WRITE :READ , + offset, dio->size, dio->error);if (likely (!ret)) { ret = dio->size;
PATCH 9/22 xfs: create file io error hooks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create hooks within XFS to deliver IO errors to callers. File I/O errors are usually rare, so we don't employ a static key here.
--- fs/xfs/xfs_file.h | 37 +++++++++ fs/xfs/xfs_mount.h | 3 + fs/xfs/xfs_aops.c | 2 fs/xfs/xfs_file.c | 174 +++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_notify_failure.c | 5 + fs/xfs/xfs_super.c | 1 6 files changed, 221 insertions(+), 1 deletion(-)
fs/xfs/xfs_file.h +37 -0
@@ -12,4 +12,41 @@ extern const struct file_operations xfs_dir_file_operations;bool xfs_is_falloc_aligned (struct xfs_inode *ip,loff_t pos,long long int len); +enum xfs_file_ioerror_type { +XFS_FILE_IOERROR_BUFFERED_READ , +XFS_FILE_IOERROR_BUFFERED_WRITE , +XFS_FILE_IOERROR_DIRECT_READ , +XFS_FILE_IOERROR_DIRECT_WRITE , +XFS_FILE_IOERROR_DATA_LOST , +}; + +struct xfs_file_ioerror_params { +xfs_ino_t ino; +loff_t pos; +u64 len; +u32 gen; +int error; +}; + +#ifdef CONFIG_XFS_LIVE_HOOKS +struct xfs_file_ioerror_hook { +struct xfs_hook ioerror_hook; +}; + +int xfs_file_ioerror_hook_add (struct xfs_mount *mp, +struct xfs_file_ioerror_hook *hook); +void xfs_file_ioerror_hook_del (struct xfs_mount *mp, +struct xfs_file_ioerror_hook *hook); +void xfs_file_ioerror_hook_setup (struct xfs_file_ioerror_hook *hook, +notifier_fn_t mod_fn); + +void xfs_vm_ioerror (struct address_space *mapping,int direction,loff_t pos, +u64 len,int error); + +void xfs_inode_media_error (struct xfs_inode *ip,loff_t pos,u64 len); +#else +# define xfs_vm_ioerror NULL +# define xfs_inode_media_error (...) ((void)0) +#endif /* CONFIG_XFS_LIVE_HOOKS */ +#endif /* __XFS_FILE_H__ */
fs/xfs/xfs_mount.h +3 -0
@@ -352,6 +352,9 @@ typedef struct xfs_mount {/* Hook to feed media error events to a daemon. */ struct xfs_hooks m_media_error_hooks; + +/* Hook to feed file io error events to a daemon. */ +struct xfs_hooks m_file_ioerror_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_aops.c +2 -0
@@ -22,6 +22,7 @@#include "xfs_icache.h" #include "xfs_zone_alloc.h" #include "xfs_rtgroup.h" +#include "xfs_file.h" struct xfs_writepage_ctx {struct iomap_writepage_ctx ctx;
@@ -810,6 +811,7 @@ const struct address_space_operations xfs_address_space_operations = { .is_partially_uptodate = iomap_is_partially_uptodate, .error_remove_folio = generic_error_remove_folio, .swap_activate = xfs_vm_swap_activate, + .ioerror = xfs_vm_ioerror, };const struct address_space_operations xfs_dax_aops = {
fs/xfs/xfs_file.c +173 -1
192 lines not on this page · raw message
fs/xfs/xfs_notify_failure.c +5 -0
@@ -22,6 +22,7 @@#include "xfs_notify_failure.h" #include "xfs_rtgroup.h" #include "xfs_rtrmap_btree.h" +#include "xfs_file.h" #include <linux/mm.h> #include <linux/dax.h>
@@ -167,6 +168,10 @@ xfs_dax_failure_fn(invalidate_inode_pages2_range (mapping, pgoff, pgoff + pgcnt -1 ); +xfs_inode_media_error (ip, +XFS_FSB_TO_B (mp, (u64 )pgoff <<PAGE_SHIFT ), +XFS_FSB_TO_B (mp, (u64 )pgcnt <<PAGE_SHIFT )); +xfs_irele (ip);return error; }
fs/xfs/xfs_super.c +1 -0
@@ -2294,6 +2294,7 @@ xfs_init_fs_context(xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks);xfs_hooks_init (&mp->m_media_error_hooks); +xfs_hooks_init (&mp->m_file_ioerror_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 10/22 xfs: create a special file to pass filesystem health to userspace
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create an ioctl that installs a file descriptor backed by an anon_inode file that will convey filesystem health events to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 8 ++ fs/xfs/xfs_healthmon.h | 16 +++++ fs/xfs/Kconfig | 8 ++ fs/xfs/Makefile | 1 fs/xfs/xfs_healthmon.c | 157 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_ioctl.c | 4 + 6 files changed, 194 insertions(+) create mode 100644 fs/xfs/xfs_healthmon.h create mode 100644 fs/xfs/xfs_healthmon.c
fs/xfs/libxfs/xfs_fs.h +8 -0
@@ -1003,6 +1003,13 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +struct xfs_health_monitor { +__u64 flags;/* flags */ +__u8 format;/* output format */ +__u8 pad1[7 ];/* zeroes */ +__u64 pad2[2 ];/* zeroes */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1042,6 +1049,7 @@ struct xfs_rtgroup_geometry {#define XFS_IOC_GETPARENTS_BY_HANDLE _IOWR('X', 63, struct xfs_getparents_by_handle)#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry) +#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (c) 2024-2025 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +#ifndef __XFS_HEALTHMON_H__ +#define __XFS_HEALTHMON_H__ + +#ifdef CONFIG_XFS_HEALTH_MONITOR +long xfs_ioc_health_monitor (struct xfs_mount *mp, +struct xfs_health_monitor __user *arg); +#else +# define xfs_ioc_health_monitor (mp, hmo) (-ENOTTY) +#endif /* CONFIG_XFS_HEALTH_MONITOR */ + +#endif /* __XFS_HEALTHMON_H__ */
fs/xfs/Kconfig +8 -0
@@ -130,6 +130,14 @@ config XFS_RT If unsure, say N. +config XFS_HEALTH_MONITOR + bool "Report filesystem health events to userspace" + depends on XFS_FS + select XFS_LIVE_HOOKS + default y + help + Report health events to userspace programs. + config XFS_DRAIN_INTENTS bool select JUMP_LABEL if HAVE_ARCH_JUMP_LABEL
fs/xfs/Makefile +1 -0
@@ -157,6 +157,7 @@ xfs-$(CONFIG_XFS_DRAIN_INTENTS) += xfs_drain.o xfs-$(CONFIG_XFS_LIVE_HOOKS) += xfs_hooks.o xfs-$(CONFIG_XFS_MEMORY_BUFS) += xfs_buf_mem.o xfs-$(CONFIG_XFS_BTREE_IN_MEM) += libxfs/xfs_btree_mem.o +xfs-$(CONFIG_XFS_HEALTH_MONITOR) += xfs_healthmon.o # online scrub/repair ifeq ($(CONFIG_XFS_ONLINE_SCRUB),y)
fs/xfs/xfs_healthmon.c new file +157 -0
157 lines not on this page · raw message
fs/xfs/xfs_ioctl.c +4 -0
@@ -41,6 +41,7 @@#include "xfs_exchrange.h" #include "xfs_handle.h" #include "xfs_rtgroup.h" +#include "xfs_healthmon.h" #include <linux/mount.h> #include <linux/fileattr.h>
@@ -1421,6 +1422,9 @@ xfs_file_ioctl(case XFS_IOC_COMMIT_RANGE :return xfs_ioc_commit_range (filp, arg); +case XFS_IOC_HEALTH_MONITOR : +return xfs_ioc_health_monitor (mp, arg); +default :return -ENOTTY ; }
PATCH 11/22 xfs: create event queuing, formatting, and discovery infrastructure
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle. Make the kernel export C structures via read().
In a previous iteration of this new subsystem, I wanted to explore data exchange formats that are more flexible and easier for humans to read than C structures. The thought being that when we want to rev (or worse, enlarge) the event format, it ought to be trivially easy to do that in a way that doesn't break old userspace.
I looked at formats such as protobufs and capnproto. These look really nice in that extending the wire format is fairly easy, you can give it a data schema and it generates the serialization code for you, handles endianness problems, etc. The huge downside is that neither support C all that well.
Too hard, and didn't want to port either of those huge sprawling libraries first to the kernel and then again to xfsprogs. Then I thought, how about JSON? Javascript objects are human readable, the kernel can emit json without much fuss (it's all just strings!) and there are plenty of interpreters for python/rust/c/etc.
There's a proposed schema format for json, which means that xfs can publish a description of the events that kernel will emit. Userspace consumers (e.g. xfsprogs/xfs_healer) can embed the same schema document and use it to validate the incoming events from the kernel, which means it can discard events that it doesn't understand, or garbage being emitted due to bugs.
However, json has a huge crutch -- javascript is well known for its vague definitions of what are numbers. This makes expressing a large number rather fraught, because the runtime is free to represent a number in nearly any way it wants. Stupider ones will truncate values to word size, others will roll out doubles for uint52_t (yes, fifty-two) with the resulting loss of precision. Not good when you're dealing with discrete units.
It just so happens that python's json library is smart enough to see a sequence of digits and put them in a u64 (at least on x86_64/aarch64) but an actual javascript interpreter (pasting into Firefox) isn't necessarily so clever.
It turns out that none of the proposed json schemas were ever ratified even in an open-consensus way, so json blobs are still just loosely structured blobs. The parsing in userspace was also noticeably slow and memory-consumptive.
Hence only the C interface survives.
--- fs/xfs/libxfs/xfs_fs.h | 47 ++++ fs/xfs/xfs_healthmon.h | 29 +++ fs/xfs/xfs_linux.h | 3 fs/xfs/xfs_trace.h | 170 +++++++++++++++ fs/xfs/xfs_healthmon.c | 542 +++++++++++++++++++++++++++++++++++++++++++++++- fs/xfs/xfs_trace.c | 2 lib/seq_buf.c | 1 7 files changed, 787 insertions(+), 7 deletions(-)
fs/xfs/libxfs/xfs_fs.h +47 -0
@@ -1003,6 +1003,45 @@ struct xfs_rtgroup_geometry {#define XFS_RTGROUP_GEOM_SICK_RMAPBT (1U << 3)/* reverse mappings */ #define XFS_RTGROUP_GEOM_SICK_REFCNTBT (1U << 4)/* reference counts */ +/* Health monitor event domains */ + +/* affects the whole fs */ +#define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) + +/* Health monitor event types */ + +/* status of the monitor itself */ +#define XFS_HEALTH_MONITOR_TYPE_RUNNING (0) +#define XFS_HEALTH_MONITOR_TYPE_LOST (1) + +/* lost events */ +struct xfs_health_monitor_lost { +__u64 count; +}; + +struct xfs_health_monitor_event { +/* XFS_HEALTH_MONITOR_DOMAIN_* */ +__u32 domain; + +/* XFS_HEALTH_MONITOR_TYPE_* */ +__u32 type; + +/* Timestamp of the event, in nanoseconds since the Unix epoch */ +__u64 time_ns; + +/* +* Details of the event. The primary clients are written in python +* and rust, so break this up because bindgen hates anonymous structs +* and unions. +*/ +union { +struct xfs_health_monitor_lost lost; + } e; + +/* zeroes */ +__u64 pad[2 ]; +}; +struct xfs_health_monitor {__u64 flags;/* flags */ __u8 format;/* output format */
@@ -1010,6 +1049,14 @@ struct xfs_health_monitor {__u64 pad2[2 ];/* zeroes */ }; +/* Return all health status events, not just deltas */ +#define XFS_HEALTH_MONITOR_VERBOSE (1ULL << 0) + +#define XFS_HEALTH_MONITOR_ALL (XFS_HEALTH_MONITOR_VERBOSE) + +/* Initial return format version */ +#define XFS_HEALTH_MONITOR_FMT_V0 (0) +/* * ioctl commands that are used by Linux filesystems */
fs/xfs/xfs_healthmon.h +29 -0
@@ -6,6 +6,35 @@#ifndef __XFS_HEALTHMON_H__#define __XFS_HEALTHMON_H__ +enum xfs_healthmon_type { +XFS_HEALTHMON_RUNNING ,/* monitor running */ +XFS_HEALTHMON_LOST ,/* message lost */ +}; + +enum xfs_healthmon_domain { +XFS_HEALTHMON_MOUNT ,/* affects the whole fs */ +}; + +struct xfs_healthmon_event { +struct xfs_healthmon_event *next; + +enum xfs_healthmon_type type; +enum xfs_healthmon_domain domain; + +uint64_t time_ns; + +union { +/* lost events */ +struct { +uint64_t lostcount; + }; +/* mount */ +struct { +unsigned int flags; + }; + }; +}; +#ifdef CONFIG_XFS_HEALTH_MONITOR long xfs_ioc_health_monitor (struct xfs_mount *mp,struct xfs_health_monitor __user *arg);
fs/xfs/xfs_linux.h +3 -0
@@ -63,6 +63,9 @@ typedef __u32 xfs_nlink_t;#include <linux/xattr.h> #include <linux/mnt_idmapping.h> #include <linux/debugfs.h> +#ifdef CONFIG_XFS_HEALTH_MONITOR +# include <linux/seq_buf.h> +#endif #include <asm/page.h> #include <asm/div64.h>
fs/xfs/xfs_trace.h +170 -0
182 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +535 -7
611 lines not on this page · raw message
fs/xfs/xfs_trace.c +2 -0
@@ -51,6 +51,8 @@#include "xfs_rtgroup.h" #include "xfs_zone_alloc.h" #include "xfs_zone_priv.h" +#include "xfs_health.h" +#include "xfs_healthmon.h" /* * We include this last to have the helpers above available for the trace
lib/seq_buf.c +1 -0
@@ -245,6 +245,7 @@ int seq_buf_putmem(struct seq_buf *s, const void *mem, unsigned int len)seq_buf_set_overflow (s);return -1 ; } +EXPORT_SYMBOL_GPL (seq_buf_putmem);#define MAX_MEMHEX_BYTES 8U#define HEX_CHARS (MAX_MEMHEX_BYTES*2 + 1)
PATCH 12/22 xfs: report metadata health events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a metadata health event hook so that we can send events to userspace as we collect information. The unmount hook severs the weak reference between the health monitor and the filesystem it's monitoring; when this happens, we stop reporting events because there's no longer any point.
--- fs/xfs/libxfs/xfs_fs.h | 38 +++++ fs/xfs/libxfs/xfs_health.h | 5 + fs/xfs/xfs_healthmon.h | 31 ++++ fs/xfs/xfs_trace.h | 98 +++++++++++++ fs/xfs/xfs_health.c | 67 +++++++++ fs/xfs/xfs_healthmon.c | 333 +++++++++++++++++++++++++++++++++++++++++++- 6 files changed, 563 insertions(+), 9 deletions(-)
fs/xfs/libxfs/xfs_fs.h +38 -0
@@ -1008,17 +1008,52 @@ struct xfs_rtgroup_geometry {/* affects the whole fs */ #define XFS_HEALTH_MONITOR_DOMAIN_MOUNT (0) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FS (1) +#define XFS_HEALTH_MONITOR_DOMAIN_AG (2) +#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3) +#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* Health monitor event types */ /* status of the monitor itself */ #define XFS_HEALTH_MONITOR_TYPE_RUNNING (0)#define XFS_HEALTH_MONITOR_TYPE_LOST (1) +/* metadata health events */ +#define XFS_HEALTH_MONITOR_TYPE_SICK (2) +#define XFS_HEALTH_MONITOR_TYPE_CORRUPT (3) +#define XFS_HEALTH_MONITOR_TYPE_HEALTHY (4) + +/* filesystem was unmounted */ +#define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* lost events */ struct xfs_health_monitor_lost {__u64 count; }; +/* fs/rt metadata */ +struct xfs_health_monitor_fs { +/* XFS_FSOP_GEOM_SICK_* flags */ +__u32 mask; +}; + +/* ag/rtgroup metadata */ +struct xfs_health_monitor_group { +/* XFS_{AG,RTGROUP}_SICK_* flags */ +__u32 mask; +__u32 gno; +}; + +/* inode metadata */ +struct xfs_health_monitor_inode { +/* XFS_BS_SICK_* flags */ +__u32 mask; +__u32 gen; +__u64 ino; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1036,6 +1071,9 @@ struct xfs_health_monitor_event {*/ union {struct xfs_health_monitor_lost lost; +struct xfs_health_monitor_fs fs; +struct xfs_health_monitor_group group; +struct xfs_health_monitor_inode inode; } e;/* zeroes */
fs/xfs/libxfs/xfs_health.h +5 -0
@@ -336,4 +336,9 @@ void xfs_health_hook_del(struct xfs_mount *mp, struct xfs_health_hook *hook);void xfs_health_hook_setup (struct xfs_health_hook *hook,notifier_fn_t mod_fn);#endif /* CONFIG_XFS_LIVE_HOOKS */ +unsigned int xfs_healthmon_inode_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_rtgroup_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_perag_mask (unsigned int sick_mask); +unsigned int xfs_healthmon_fs_mask (unsigned int sick_mask); +#endif /* __XFS_HEALTH_H__ */
fs/xfs/xfs_healthmon.h +31 -0
@@ -9,10 +9,23 @@enum xfs_healthmon_type {XFS_HEALTHMON_RUNNING ,/* monitor running */ XFS_HEALTHMON_LOST ,/* message lost */ +XFS_HEALTHMON_UNMOUNT ,/* filesystem is unmounting */ + +/* metadata health events */ +XFS_HEALTHMON_SICK ,/* runtime corruption observed */ +XFS_HEALTHMON_CORRUPT ,/* fsck reported corruption */ +XFS_HEALTHMON_HEALTHY ,/* fsck reported healthy structure */ + };enum xfs_healthmon_domain {XFS_HEALTHMON_MOUNT ,/* affects the whole fs */ + +/* metadata health events */ +XFS_HEALTHMON_FS ,/* main filesystem metadata */ +XFS_HEALTHMON_AG ,/* allocation group metadata */ +XFS_HEALTHMON_INODE ,/* inode metadata */ +XFS_HEALTHMON_RTGROUP ,/* realtime group metadata */ };struct xfs_healthmon_event {
@@ -32,6 +45,24 @@ struct xfs_healthmon_event {struct {unsigned int flags; }; +/* fs/rt metadata */ +struct { +/* XFS_SICK_* flags */ +unsigned int fsmask; + }; +/* ag/rtgroup metadata */ +struct { +/* XFS_SICK_* flags */ +unsigned int grpmask; +unsigned int group; + }; +/* inode metadata */ +struct { +/* XFS_SICK_INO_* flags */ +unsigned int imask; +uint32_t gen; +xfs_ino_t ino; + }; }; };
fs/xfs/xfs_trace.h +96 -2
127 lines not on this page · raw message
fs/xfs/xfs_health.c +67 -0
91 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +326 -7
447 lines not on this page · raw message
PATCH 13/22 xfs: report shutdown events through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a shutdown hook so that we can send notifications to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 18 +++++++++ fs/xfs/xfs_healthmon.h | 5 ++- fs/xfs/xfs_trace.h | 28 ++++++++++++++ fs/xfs/xfs_healthmon.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 141 insertions(+), 3 deletions(-)
fs/xfs/libxfs/xfs_fs.h +18 -0
@@ -1028,6 +1028,9 @@ struct xfs_rtgroup_geometry {/* filesystem was unmounted */ #define XFS_HEALTH_MONITOR_TYPE_UNMOUNT (5) +/* filesystem shutdown */ +#define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1054,6 +1057,20 @@ struct xfs_health_monitor_inode {__u64 ino; }; +/* shutdown reasons */ +#define XFS_HEALTH_SHUTDOWN_META_IO_ERROR (1u << 0) +#define XFS_HEALTH_SHUTDOWN_LOG_IO_ERROR (1u << 1) +#define XFS_HEALTH_SHUTDOWN_FORCE_UMOUNT (1u << 2) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_INCORE (1u << 3) +#define XFS_HEALTH_SHUTDOWN_CORRUPT_ONDISK (1u << 4) +#define XFS_HEALTH_SHUTDOWN_DEVICE_REMOVED (1u << 5) + +/* shutdown */ +struct xfs_health_monitor_shutdown { +/* XFS_HEALTH_SHUTDOWN_* flags */ +__u32 reasons; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1074,6 +1091,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_fs fs;struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode; +struct xfs_health_monitor_shutdown shutdown; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +4 -1
@@ -11,6 +11,9 @@ enum xfs_healthmon_type {XFS_HEALTHMON_LOST ,/* message lost */ XFS_HEALTHMON_UNMOUNT ,/* filesystem is unmounting */ +/* filesystem shutdown */ +XFS_HEALTHMON_SHUTDOWN , +/* metadata health events */ XFS_HEALTHMON_SICK ,/* runtime corruption observed */ XFS_HEALTHMON_CORRUPT, /* fsck reported corruption */
@@ -41,7 +44,7 @@ struct xfs_healthmon_event {struct {uint64_t lostcount; }; -/* mount */ +/* shutdown */ struct {unsigned int flags; };
fs/xfs/xfs_trace.h +28 -0
@@ -6009,8 +6009,32 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_read_finish);DEFINE_HEALTHMON_EVENT (xfs_healthmon_release);DEFINE_HEALTHMON_EVENT (xfs_healthmon_unmount); +TRACE_EVENT (xfs_healthmon_shutdown_hook, + TP_PROTO(const struct xfs_mount *mp, uint32_t shutdown_flags, + unsigned int events, unsigned long long lost_prev), +TP_ARGS (mp, shutdown_flags, events, lost_prev), +TP_STRUCT__entry ( + __field(dev_t, dev) + __field(uint32_t, shutdown_flags) + __field(unsigned int, events) + __field(unsigned long long, lost_prev) + ), +TP_fast_assign ( + __entry->dev = mp ? mp->m_super->s_dev : 0; + __entry->shutdown_flags = shutdown_flags; + __entry->events = events; + __entry->lost_prev = lost_prev; + ), +TP_printk ("dev %d:%d shutdown_flags %s events %u lost_prev? %llu" , +MAJOR (__entry->dev),MINOR (__entry->dev), +__print_flags (__entry->shutdown_flags,"|" ,XFS_SHUTDOWN_STRINGS ), + __entry->events, + __entry->lost_prev) +); +#define XFS_HEALTHMON_TYPE_STRINGS \ { XFS_HEALTHMON_LOST, "lost" }, \ + { XFS_HEALTHMON_SHUTDOWN, "shutdown" }, \ { XFS_HEALTHMON_UNMOUNT, "unmount" }, \ { XFS_HEALTHMON_SICK, "sick" }, \ { XFS_HEALTHMON_CORRUPT, "corrupt" }, \
@@ -6024,6 +6048,7 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_unmount); { XFS_HEALTHMON_RTGROUP, "rtgroup" }TRACE_DEFINE_ENUM (XFS_HEALTHMON_LOST ); +TRACE_DEFINE_ENUM (XFS_HEALTHMON_SHUTDOWN );TRACE_DEFINE_ENUM (XFS_HEALTHMON_UNMOUNT );TRACE_DEFINE_ENUM (XFS_HEALTHMON_SICK );TRACE_DEFINE_ENUM (XFS_HEALTHMON_CORRUPT );
@@ -6064,6 +6089,9 @@ DECLARE_EVENT_CLASS(xfs_healthmon_event_class,switch (__entry->domain) {case XFS_HEALTHMON_MOUNT :switch (__entry->type) { +case XFS_HEALTHMON_SHUTDOWN : + __entry->mask = event->flags; +break ;case XFS_HEALTHMON_LOST : __entry->lostcount = event->lostcount;break ;
fs/xfs/xfs_healthmon.c +91 -2
158 lines not on this page · raw message
PATCH 14/22 xfs: report media errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have hooks to report media errors, connect this to the health monitor as well.
--- fs/xfs/libxfs/xfs_fs.h | 15 +++++++++ fs/xfs/xfs_healthmon.h | 12 +++++++ fs/xfs/xfs_trace.h | 57 ++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_healthmon.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++- fs/xfs/xfs_trace.c | 1 + 5 files changed, 160 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +15 -0
@@ -1014,6 +1014,11 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_INODE (3)#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP (4) +/* disk events */ +#define XFS_HEALTH_MONITOR_DOMAIN_DATADEV (5) +#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6) +#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1031,6 +1036,9 @@ struct xfs_rtgroup_geometry {/* filesystem shutdown */ #define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN (6) +/* media errors */ +#define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1071,6 +1079,12 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* disk media errors */ +struct xfs_health_monitor_media { +__u64 daddr; +__u64 bbcount; +}; +struct xfs_health_monitor_event {/* XFS_HEALTH_MONITOR_DOMAIN_* */ __u32 domain;
@@ -1092,6 +1106,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_group group;struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown; +struct xfs_health_monitor_media media; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +12 -0
@@ -19,6 +19,8 @@ enum xfs_healthmon_type {XFS_HEALTHMON_CORRUPT ,/* fsck reported corruption */ XFS_HEALTHMON_HEALTHY ,/* fsck reported healthy structure */ +/* media errors */ + XFS_HEALTHMON_MEDIA_ERROR, }; enum xfs_healthmon_domain {
@@ -29,6 +31,11 @@ enum xfs_healthmon_domain {XFS_HEALTHMON_AG ,/* allocation group metadata */ XFS_HEALTHMON_INODE ,/* inode metadata */ XFS_HEALTHMON_RTGROUP ,/* realtime group metadata */ + +/* media errors */ +XFS_HEALTHMON_DATADEV , +XFS_HEALTHMON_RTDEV , + XFS_HEALTHMON_LOGDEV, }; struct xfs_healthmon_event {
@@ -66,6 +73,11 @@ struct xfs_healthmon_event {uint32_t gen;xfs_ino_t ino; }; +/* media errors */ +struct { +xfs_daddr_t daddr; +uint64_t bbcount; + }; }; };
fs/xfs/xfs_trace.h +57 -0
75 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +75 -2
142 lines not on this page · raw message
fs/xfs/xfs_trace.c +1 -0
@@ -53,6 +53,7 @@#include "xfs_zone_priv.h" #include "xfs_health.h" #include "xfs_healthmon.h" +#include "xfs_notify_failure.h" /* * We include this last to have the helpers above available for the trace
PATCH 15/22 xfs: report file io errors through healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Set up a file io error event hook so that we can send events about read errors, writeback errors, and directio errors to userspace.
--- fs/xfs/libxfs/xfs_fs.h | 19 +++++++++ fs/xfs/xfs_healthmon.h | 17 ++++++++ fs/xfs/xfs_trace.h | 59 +++++++++++++++++++++++++++++ fs/xfs/xfs_healthmon.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++- fs/xfs/xfs_trace.c | 1 5 files changed, 192 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +19 -0
@@ -1019,6 +1019,9 @@ struct xfs_rtgroup_geometry {#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV (6)#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_DOMAIN_FILERANGE (8) +/* Health monitor event types */ /* status of the monitor itself */
@@ -1039,6 +1042,13 @@ struct xfs_rtgroup_geometry {/* media errors */ #define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR (7) +/* file range events */ +#define XFS_HEALTH_MONITOR_TYPE_BUFREAD (8) +#define XFS_HEALTH_MONITOR_TYPE_BUFWRITE (9) +#define XFS_HEALTH_MONITOR_TYPE_DIOREAD (10) +#define XFS_HEALTH_MONITOR_TYPE_DIOWRITE (11) +#define XFS_HEALTH_MONITOR_TYPE_DATALOST (12) +/* lost events */ struct xfs_health_monitor_lost {__u64 count;
@@ -1079,6 +1089,14 @@ struct xfs_health_monitor_shutdown {__u32 reasons; }; +/* file range events */ +struct xfs_health_monitor_filerange { +__u64 pos; +__u64 len; +__u64 ino; +__u32 gen; +}; +/* disk media errors */ struct xfs_health_monitor_media {__u64 daddr;
@@ -1107,6 +1125,7 @@ struct xfs_health_monitor_event {struct xfs_health_monitor_inode inode;struct xfs_health_monitor_shutdown shutdown;struct xfs_health_monitor_media media; +struct xfs_health_monitor_filerange filerange; } e;/* zeroes */
fs/xfs/xfs_healthmon.h +17 -0
@@ -21,6 +21,13 @@ enum xfs_healthmon_type {/* media errors */ XFS_HEALTHMON_MEDIA_ERROR , + +/* file range events */ +XFS_HEALTHMON_BUFREAD , +XFS_HEALTHMON_BUFWRITE , +XFS_HEALTHMON_DIOREAD , +XFS_HEALTHMON_DIOWRITE , + XFS_HEALTHMON_DATALOST, }; enum xfs_healthmon_domain {
@@ -36,6 +43,9 @@ enum xfs_healthmon_domain {XFS_HEALTHMON_DATADEV ,XFS_HEALTHMON_RTDEV ,XFS_HEALTHMON_LOGDEV , + +/* file range events */ + XFS_HEALTHMON_FILERANGE, }; struct xfs_healthmon_event {
@@ -78,6 +88,13 @@ struct xfs_healthmon_event {xfs_daddr_t daddr;uint64_t bbcount; }; +/* file range events */ +struct { +xfs_ino_t fino; +loff_t fpos; +uint64_t flen; +uint32_t fgen; + }; }; };
fs/xfs/xfs_trace.h +59 -0
77 lines not on this page · raw message
fs/xfs/xfs_healthmon.c +96 -2
163 lines not on this page · raw message
fs/xfs/xfs_trace.c +1 -0
@@ -54,6 +54,7 @@#include "xfs_health.h" #include "xfs_healthmon.h" #include "xfs_notify_failure.h" +#include "xfs_file.h" /* * We include this last to have the helpers above available for the trace
PATCH 16/22 xfs: allow reconfiguration of the health monitoring device
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that we can reconfigure the health monitoring device by calling the XFS_IOC_HEALTH_MONITOR ioctl on it.
--- fs/xfs/xfs_healthmon.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
fs/xfs/xfs_healthmon.c +44 -0
@@ -23,6 +23,7 @@#include "xfs_fsops.h" #include "xfs_notify_failure.h" #include "xfs_file.h" +#include "xfs_ioctl.h" #include <linux/anon_inodes.h> #include <linux/eventpoll.h>
@@ -1140,6 +1141,48 @@ xfs_healthmon_show_fdinfo( }#endif +/* Reconfigure the health monitor. */ +STATIC long +xfs_healthmon_reconfigure ( +struct file *file, +unsigned int cmd, +void __user *arg) +{ +struct xfs_health_monitor hmo; +struct xfs_healthmon *hm = file->private_data; + +if (copy_from_user (&hmo, arg,sizeof (hmo))) +return -EFAULT ; + +if (!xfs_healthmon_validate (&hmo)) +return -EINVAL ; + +mutex_lock (&hm->lock); + hm->format = hmo.format; + hm->verbose = !!(hmo.flags &XFS_HEALTH_MONITOR_VERBOSE ); +mutex_unlock (&hm->lock); +return 0 ; +} + +/* Handle ioctls for the health monitoring thread. */ +STATIC long +xfs_healthmon_ioctl ( +struct file *file, +unsigned int cmd, +unsigned long p) +{ +void __user *arg = (void __user *)p; + +switch (cmd) { +case XFS_IOC_HEALTH_MONITOR : +return xfs_healthmon_reconfigure (file, cmd, arg); +default : +break ; + } + +return -ENOTTY ; +} +static const struct file_operations xfs_healthmon_fops = { .owner =THIS_MODULE , #ifdef CONFIG_PROC_FS
@@ -1148,6 +1191,7 @@ static const struct file_operations xfs_healthmon_fops = { .read_iter = xfs_healthmon_read_iter, .poll = xfs_healthmon_poll, .release = xfs_healthmon_release, + .unlocked_ioctl = xfs_healthmon_ioctl, };/*
PATCH 17/22 xfs: validate fds against running healthmon
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create a new ioctl for the healthmon file that checks that a given fd points to the same filesystem that the healthmon file is monitoring. This allows xfs_healer to check that when it reopens a mountpoint to perform repairs, the file that it gets matches the filesystem that generated the corruption report.
(Note that xfs_healer doesn't maintain an open fd to a filesystem that it's monitoring so that it doesn't pin the mount.)
--- fs/xfs/libxfs/xfs_fs.h | 10 ++++++++++ fs/xfs/xfs_healthmon.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+)
fs/xfs/libxfs/xfs_fs.h +10 -0
@@ -1147,6 +1147,15 @@ struct xfs_health_monitor {/* Initial return format version */ #define XFS_HEALTH_MONITOR_FMT_V0 (0) +/* +* Check that a given fd points to the same filesystem that the health monitor +* is monitoring. +*/ +struct xfs_health_samefs { +__s32 fd; +__u32 flags;/* zero for now */ +}; +/* * ioctl commands that are used by Linux filesystems */
@@ -1187,6 +1196,7 @@ struct xfs_health_monitor {#define XFS_IOC_SCRUBV_METADATA _IOWR('X', 64, struct xfs_scrub_vec_head)#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor) +#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.c +32 -0
@@ -1164,6 +1164,36 @@ xfs_healthmon_reconfigure(return 0 ; } +/* Does the fd point to the same filesystem as the one we're monitoring? */ +STATIC long +xfs_healthmon_samefs ( +struct file *file, +unsigned int cmd, +void __user *arg) +{ +struct xfs_health_samefs hms; +struct xfs_healthmon *hm = file->private_data; +struct inode *hms_inode; +int ret =0 ; + +if (copy_from_user (&hms, arg,sizeof (hms))) +return -EFAULT ; + +if (hms.flags) +return -EINVAL ; + +CLASS (fd, hms_fd)(hms.fd); +if (fd_empty (hms_fd)) +return -EBADF ; + + hms_inode =file_inode (fd_file (hms_fd)); +mutex_lock (&hm->lock); +if (!hm->mp || hm->mp->m_super != hms_inode->i_sb) + ret = -ESTALE ; +mutex_unlock (&hm->lock); +return ret; +} +/* Handle ioctls for the health monitoring thread. */ STATIC long xfs_healthmon_ioctl(
@@ -1176,6 +1206,8 @@ xfs_healthmon_ioctl(switch (cmd) {case XFS_IOC_HEALTH_MONITOR :return xfs_healthmon_reconfigure (file, cmd, arg); +case XFS_IOC_HEALTH_SAMEFS : +return xfs_healthmon_samefs (file, cmd, arg);default :break ; }
PATCH 18/22 xfs: add media error reporting ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new privileged ioctl so that xfs_scrub can report media errors to the kernel for further processing.
--- fs/xfs/libxfs/xfs_fs.h | 16 ++++ fs/xfs/xfs_notify_failure.h | 8 ++ fs/xfs/xfs_trace.h | 2 fs/xfs/Makefile | 6 - fs/xfs/xfs_healthmon.c | 2 fs/xfs/xfs_ioctl.c | 3 + fs/xfs/xfs_notify_failure.c | 187 +++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 213 insertions(+), 11 deletions(-)
fs/xfs/libxfs/xfs_fs.h +16 -0
@@ -1156,6 +1156,21 @@ struct xfs_health_samefs {__u32 flags;/* zero for now */ }; +/* Report a media error */ +struct xfs_media_error { +__u64 flags;/* flags */ +__u64 daddr;/* disk address of range */ +__u64 bbcount;/* length, in 512b blocks */ +__u64 pad;/* zero */ +}; + +#define XFS_MEDIA_ERROR_DATADEV (1)/* data device */ +#define XFS_MEDIA_ERROR_LOGDEV (2)/* external log device */ +#define XFS_MEDIA_ERROR_RTDEV (3)/* realtime device */ + +/* bottom byte of flags is the device code */ +#define XFS_MEDIA_ERROR_DEVMASK (0xFF) +/* * ioctl commands that are used by Linux filesystems */
@@ -1197,6 +1212,7 @@ struct xfs_health_samefs {#define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)#define XFS_IOC_HEALTH_MONITOR _IOW ('X', 68, struct xfs_health_monitor)#define XFS_IOC_HEALTH_SAMEFS _IOW ('X', 69, struct xfs_health_samefs) +#define XFS_IOC_MEDIA_ERROR _IOW ('X', 70, struct xfs_media_error)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_notify_failure.h +7 -1
@@ -6,7 +6,9 @@#ifndef __XFS_NOTIFY_FAILURE_H__#define __XFS_NOTIFY_FAILURE_H__ +#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX )extern const struct dax_holder_operations xfs_dax_holder_operations; +#endif enum xfs_failed_device { XFS_FAILED_DATADEV,
@@ -14,7 +16,7 @@ enum xfs_failed_device { XFS_FAILED_RTDEV, }; -#if defined(CONFIG_XFS_LIVE_HOOKS) && defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX) +#if defined(CONFIG_XFS_LIVE_HOOKS) struct xfs_media_error_params { struct xfs_mount *mp; enum xfs_failed_device fdev;
@@ -41,4 +43,8 @@ struct xfs_media_error_hook { };# define xfs_media_error_hook_setup (...) ((void)0)#endif /* CONFIG_XFS_LIVE_HOOKS */ +struct xfs_media_error ; +int xfs_ioc_media_error (struct xfs_mount *mp, +struct xfs_media_error __user *arg); +#endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_trace.h +0 -2
@@ -6213,7 +6213,6 @@ TRACE_EVENT(xfs_healthmon_metadata_hook, __entry->lost_prev) ); -#if defined(CONFIG_XFS_LIVE_HOOKS ) && defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX ) TRACE_EVENT(xfs_healthmon_media_error_hook, TP_PROTO(const struct xfs_media_error_params *p, unsigned int events, unsigned long long lost_prev),
@@ -6261,7 +6260,6 @@ TRACE_EVENT(xfs_healthmon_media_error_hook, __entry->events, __entry->lost_prev) ); -#endif #define XFS_FILE_IOERROR_STRINGS \ { XFS_FILE_IOERROR_BUFFERED_READ, "readahead" }, \
fs/xfs/Makefile +1 -5
@@ -98,6 +98,7 @@ xfs-y += xfs_aops.o \ xfs_message.o \ xfs_mount.o \ xfs_mru_cache.o \ + xfs_notify_failure.o \ xfs_pwork.o \ xfs_reflink.o \ xfs_stats.o \
@@ -148,11 +149,6 @@ xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o xfs-$(CONFIG_EXPORTFS_BLOCK_OPS) += xfs_pnfs.o -# notify failure -ifeq ($(CONFIG_MEMORY_FAILURE),y) -xfs-$(CONFIG_FS_DAX) += xfs_notify_failure.o -endif - xfs-$(CONFIG_XFS_DRAIN_INTENTS) += xfs_drain.o xfs-$(CONFIG_XFS_LIVE_HOOKS) += xfs_hooks.o xfs-$(CONFIG_XFS_MEMORY_BUFS) += xfs_buf_mem.o
fs/xfs/xfs_healthmon.c +0 -2
@@ -523,7 +523,6 @@ xfs_healthmon_shutdown_hook(return NOTIFY_DONE ; } -#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_FS_DAX )/* Add a media error event to the reporting queue. */ STATIC int xfs_healthmon_media_error_hook(
@@ -574,7 +573,6 @@ xfs_healthmon_media_error_hook(mutex_unlock (&hm->lock);return NOTIFY_DONE ; } -#endif /* Add a file io error event to the reporting queue. */ STATIC int
fs/xfs/xfs_ioctl.c +3 -0
@@ -42,6 +42,7 @@#include "xfs_handle.h" #include "xfs_rtgroup.h" #include "xfs_healthmon.h" +#include "xfs_notify_failure.h" #include <linux/mount.h> #include <linux/fileattr.h>
@@ -1424,6 +1425,8 @@ xfs_file_ioctl(case XFS_IOC_HEALTH_MONITOR :return xfs_ioc_health_monitor (mp, arg); +case XFS_IOC_MEDIA_ERROR : +return xfs_ioc_media_error (mp, arg);default :return -ENOTTY ;
fs/xfs/xfs_notify_failure.c +186 -1
198 lines not on this page · raw message
PATCH 19/22 xfs: send uevents when major filesystem events happen
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Send uevents when we mount, unmount, and shut down the filesystem, so that we can trigger systemd services when major events happen.
--- fs/xfs/xfs_super.h | 13 +++++++ fs/xfs/xfs_fsops.c | 18 ++++++++++ fs/xfs/xfs_super.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 125 insertions(+)
fs/xfs/xfs_super.h +13 -0
@@ -101,4 +101,17 @@ extern struct workqueue_struct *xfs_discard_wq;struct dentry *xfs_debugfs_mkdir (const char *name,struct dentry *parent); +#define XFS_UEVENT_BUFLEN ( \ + sizeof("SID=") + sizeof_field(struct super_block, s_id) + \ + sizeof("UUID=") + UUID_STRING_LEN + \ + sizeof("META_UUID=") + UUID_STRING_LEN) + +#define XFS_UEVENT_STR_PTRS \ + NULL,/* sid */ \ +NULL ,/* uuid */ \ + NULL /* metauuid */ + +int xfs_format_uevent_strings (struct xfs_mount *mp,char *buf,ssize_t buflen, +char **env); +#endif /* __XFS_SUPER_H__ */
fs/xfs/xfs_fsops.c +18 -0
@@ -522,6 +522,23 @@ xfs_shutdown_hook_setup(# define xfs_shutdown_hook (...) ((void)0)#endif /* CONFIG_XFS_LIVE_HOOKS */ +static void +xfs_send_shutdown_uevent ( +struct xfs_mount *mp) +{ +char buf[XFS_UEVENT_BUFLEN ]; +char *env[] = { +"TYPE=shutdown" , +XFS_UEVENT_STR_PTRS , +NULL , + }; +int error; + + error =xfs_format_uevent_strings (mp, buf,sizeof (buf), &env[2 ]); +if (!error) +kobject_uevent_env (&mp->m_kobj.kobject,KOBJ_OFFLINE , env); +} +/* * Force a shutdown of the filesystem instantly while keeping the filesystem * consistent. We don't do an unmount here; just shutdown the shop, make sure
@@ -572,6 +589,7 @@ xfs_do_force_shutdown( }trace_xfs_force_shutdown (mp, tag, flags, fname, lnnum); +xfs_send_shutdown_uevent (mp); xfs_alert_tag(mp, tag, "%s (0x%x) detected at %pS (%s:%d). Shutting down filesystem.",
fs/xfs/xfs_super.c +94 -0
124 lines not on this page · raw message
PATCH 20/22 xfs: merge health monitoring events when possible
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Reduce memory consumption and event traffic by merging healthmon events whenever we actually add an event to the queue.
--- fs/xfs/xfs_trace.h | 1 fs/xfs/xfs_healthmon.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+)
fs/xfs/xfs_trace.h +1 -0
@@ -6143,6 +6143,7 @@ DEFINE_EVENT(xfs_healthmon_event_class, name, \ TP_PROTO(const struct xfs_mount *mp, const struct xfs_healthmon_event *event), \ TP_ARGS(mp, event)) DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_insert); +DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_merge);DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_push);DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_pop);DEFINE_HEALTHMONEVENT_EVENT (xfs_healthmon_format);
fs/xfs/xfs_healthmon.c +107 -0
125 lines not on this page · raw message
PATCH 21/22 xfs: restrict healthmon users further
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Because health monitoring events include file handles and deep information about the filesystem structure, restrict usage to healthmon to processes that can open the root directory and run in the initial user namespace.
--- fs/xfs/xfs_healthmon.h | 4 ++-- fs/xfs/xfs_healthmon.c | 9 ++++++++- fs/xfs/xfs_ioctl.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-)
fs/xfs/xfs_healthmon.h +2 -2
@@ -99,10 +99,10 @@ struct xfs_healthmon_event { };#ifdef CONFIG_XFS_HEALTH_MONITOR -long xfs_ioc_health_monitor (struct xfs_mount *mp, +long xfs_ioc_health_monitor (struct file *file,struct xfs_health_monitor __user *arg);#else -# define xfs_ioc_health_monitor (mp, hmo) (-ENOTTY) +# define xfs_ioc_health_monitor (file, hmo) (-ENOTTY)#endif /* CONFIG_XFS_HEALTH_MONITOR */ #endif /* __XFS_HEALTHMON_H__ */
fs/xfs/xfs_healthmon.c +8 -1
@@ -1337,18 +1337,25 @@ static const struct file_operations xfs_healthmon_fops = {*/ long xfs_ioc_health_monitor ( -struct xfs_mount *mp, +struct file *file,struct xfs_health_monitor __user *arg) {struct xfs_health_monitor hmo;struct xfs_healthmon *hm;struct xfs_healthmon_event *event; +struct xfs_inode *ip =XFS_I (file_inode (file)); +struct xfs_mount *mp = ip->i_mount;int fd;int ret;if (!capable (CAP_SYS_ADMIN ))return -EPERM ; +if (ip->i_ino != mp->m_sb.sb_rootino) +return -EPERM ; +if (current_user_ns () != &init_user_ns) +return -EPERM ; +if (copy_from_user (&hmo, arg,sizeof (hmo)))return -EFAULT ;
fs/xfs/xfs_ioctl.c +1 -1
@@ -1424,7 +1424,7 @@ xfs_file_ioctl(return xfs_ioc_commit_range (filp, arg);case XFS_IOC_HEALTH_MONITOR : -return xfs_ioc_health_monitor (mp, arg); +return xfs_ioc_health_monitor (filp, arg);case XFS_IOC_MEDIA_ERROR :return xfs_ioc_media_error (mp, arg);
PATCH 22/22 xfs: charge healthmon event objects to the memcg of the listening process
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Event objects are created in the context of whichever process experienced the health event, which means that we currently charge that process' memory cgroup controller for that object. This isn't entirely fair to that process, because it's being charged for memory that solely benefits whatever's using the healthmon fd (xfs_healer).
Therefore, save the memcg that was in place when the healthmon fd was created (which we assume is xfs_healer) and make sure the objects are charged to that memcg. This also enables sysadmins to constrain the kernel memory usage of xfs_healer through memcgs.
--- fs/xfs/xfs_healthmon.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
fs/xfs/xfs_healthmon.c +18 -0
@@ -67,6 +67,9 @@ struct xfs_healthmon {struct xfs_healthmon_event *first_event;struct xfs_healthmon_event *last_event; +/* charge event object usage to this memory cgroup */ +struct mem_cgroup *memcg; +/* live update hooks */ struct xfs_shutdown_hook shook;struct xfs_health_hook hhook;
@@ -500,6 +503,7 @@ xfs_healthmon_metadata_hook(struct xfs_health_update_params *hup = data;struct xfs_healthmon *hm;struct xfs_healthmon_event *event; +struct mem_cgroup *old_memcg;enum xfs_health_update_type type = action;unsigned int mask =0 ;int error;
@@ -511,6 +515,7 @@ xfs_healthmon_metadata_hook(return NOTIFY_DONE ;mutex_lock (&hm->lock); + old_memcg =set_active_memcg (hm->memcg);trace_xfs_healthmon_metadata_hook (hm->mp, action, hup, hm->events, hm->lost_prev_event);
@@ -586,6 +591,7 @@ xfs_healthmon_metadata_hook( goto out_event; out_unlock: +set_active_memcg (old_memcg);mutex_unlock (&hm->lock);return NOTIFY_DONE ; out_event:
@@ -602,11 +608,13 @@ xfs_healthmon_shutdown_hook( {struct xfs_healthmon *hm;struct xfs_healthmon_event *event; +struct mem_cgroup *old_memcg;int error; hm = container_of(nb, struct xfs_healthmon, shook.shutdown_hook.nb);mutex_lock (&hm->lock); + old_memcg =set_active_memcg (hm->memcg);trace_xfs_healthmon_shutdown_hook (hm->mp, action, hm->events, hm->lost_prev_event);
@@ -626,6 +634,7 @@ xfs_healthmon_shutdown_hook(kfree (event); out_unlock: +set_active_memcg (old_memcg);mutex_unlock (&hm->lock);return NOTIFY_DONE ; }
@@ -640,12 +649,14 @@ xfs_healthmon_media_error_hook(struct xfs_healthmon *hm;struct xfs_healthmon_event *event;struct xfs_media_error_params *p = data; +struct mem_cgroup *old_memcg;enum xfs_healthmon_domain domain =0 ;/* shut up gcc */ int error; hm = container_of(nb, struct xfs_healthmon, mhook.error_hook.nb);mutex_lock (&hm->lock); + old_memcg =set_active_memcg (hm->memcg);trace_xfs_healthmon_media_error_hook (p, hm->events, hm->lost_prev_event);
@@ -677,6 +688,7 @@ xfs_healthmon_media_error_hook(kfree (event); out_unlock: +set_active_memcg (old_memcg);mutex_unlock (&hm->lock);return NOTIFY_DONE ; }
@@ -691,6 +703,7 @@ xfs_healthmon_file_ioerror_hook(struct xfs_healthmon *hm;struct xfs_healthmon_event *event;struct xfs_file_ioerror_params *p = data; +struct mem_cgroup *old_memcg;enum xfs_healthmon_type type =0 ;int error;
@@ -709,6 +722,7 @@ xfs_healthmon_file_ioerror_hook( }mutex_lock (&hm->lock); + old_memcg =set_active_memcg (hm->memcg);trace_xfs_healthmon_file_ioerror_hook (hm->mp, action, p, hm->events, hm->lost_prev_event);
@@ -748,6 +762,7 @@ xfs_healthmon_file_ioerror_hook(kfree (event); out_unlock: +set_active_memcg (old_memcg);mutex_unlock (&hm->lock);return NOTIFY_DONE ; }
@@ -1188,6 +1203,7 @@ xfs_healthmon_release(xfs_healthmon_free_events (hm);if (hm->outbuf.size)kfree (hm->outbuf.buffer); +mem_cgroup_put (hm->memcg);kfree (hm);return 0 ;
@@ -1367,6 +1383,7 @@ xfs_ioc_health_monitor(return -ENOMEM ; hm->mp = mp; hm->format = hmo.format; + hm->memcg =get_mem_cgroup_from_mm (current->mm);/* * Since we already got a ref to the module, take a reference to the
@@ -1443,6 +1460,7 @@ xfs_ioc_health_monitor(xfs_health_hook_disable ();mutex_destroy (&hm->lock);xfs_healthmon_free_events (hm); +mem_cgroup_put (hm->memcg);kfree (hm);return ret; }
PATCH v3 2/2 iomap: generic file IO error reporting
In reply to Darrick J. WongHi all,
Refactor the iomap file I/O error handling code so that failures are reported in a generic way to fsnotify. Then connect the XFS health reporting to the same fsnotify, and now XFS can fsnotify userspace of all manner of problems.
This series is much more experimental than the main healer patchset, and I'd rather it not become a blocker for the main patchset. I wouldn't mind rebasing if they went in at the same time though.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=iomap-ioerr-reporting
---
Commits in this patchset:
* iomap: report file IO errors to fsnotify
* xfs: switch healthmon to use the iomap I/O error reporting
* xfs: port notify-failure to use the new vfs io error reporting
* xfs: remove file I/O error hooks
* iomap: remove I/O error hooks
* xfs: report fs metadata errors via fsnotify
--- fs/xfs/xfs_file.h | 37 -------- fs/xfs/xfs_mount.h | 3 - fs/xfs/xfs_trace.h | 35 ++++--- include/linux/fs.h | 68 ++++++++++++++ include/linux/iomap.h | 2 Documentation/filesystems/vfs.rst | 7 - fs/iomap/buffered-io.c | 8 +- fs/iomap/direct-io.c | 9 +- fs/super.c | 53 +++++++++++ fs/xfs/xfs_aops.c | 2 fs/xfs/xfs_file.c | 174 ------------------------------------- fs/xfs/xfs_health.c | 12 ++- fs/xfs/xfs_healthmon.c | 51 ++++++----- fs/xfs/xfs_notify_failure.c | 9 +- fs/xfs/xfs_super.c | 1 15 files changed, 190 insertions(+), 281 deletions(-)
PATCH 1/6 iomap: report file IO errors to fsnotify
From: Darrick J. Wong <[email protected]>
Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.
--- include/linux/fs.h | 64 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/iomap/buffered-io.c | 6 +++++ fs/iomap/direct-io.c | 5 ++++ fs/super.c | 53 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 128 insertions(+)
include/linux/fs.h +64 -0
@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );
@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error); +} + +static inline void inode_error (struct inode *inode,enum fs_error_type type, +loff_t pos,u64 len,int error) +{ +__sb_error (inode->i_sb, inode, type, pos, len, error); +} +#endif /* _LINUX_FS_H */
fs/iomap/buffered-io.c +6 -0
@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,inline void iomap_mapping_ioerror (struct address_space *mapping,int direction,loff_t pos,u64 len,int error) { +struct inode *inode = mapping->host; + +inode_error (inode, + direction ==READ ?FSERR_READAHEAD :FSERR_WRITEBACK , + pos, len, error); +if (mapping && mapping->a_ops->ioerror) mapping->a_ops->ioerror (mapping, direction, pos, len, error);
fs/iomap/direct-io.c +5 -0
@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)if (dops && dops->end_io) ret = dops->end_io (iocb, dio->size, ret, dio->flags); +if (dio->error) +inode_error (file_inode (iocb->ki_filp), + (dio->flags &IOMAP_DIO_WRITE ) ?FSERR_DIO_WRITE : +FSERR_DIO_READ , + offset, dio->size, dio->error);if (dio->error && dops && dops->ioerror) dops->ioerror(file_inode(iocb->ki_filp), (dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
fs/super.c +53 -0
62 lines not on this page · raw message
Re: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 7 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.
Suggested-by Christoph Hellwig <[email protected]>Signed-off-by "Darrick J. Wong" <[email protected]>
Looks good to me. Feel free to add:
Honza
Quoted 197 lines
--- include/linux/fs.h | 64 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/iomap/buffered-io.c | 6 +++++ fs/iomap/direct-io.c | 5 ++++ fs/super.c | 53 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 128 insertions(+)include/linux/fs.h +64
-0@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error); +} + +static inline void inode_error (struct inode *inode,enum fs_error_type type, +loff_t pos,u64 len,int error) +{ +__sb_error (inode->i_sb, inode, type, pos, len, error); +} +#endif /* _LINUX_FS_H */ fs/iomap/buffered-io.c +6
-0@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,inline void iomap_mapping_ioerror (struct address_space *mapping,int direction,loff_t pos,u64 len,int error) { +struct inode *inode = mapping->host; + +inode_error (inode, + direction ==READ ?FSERR_READAHEAD :FSERR_WRITEBACK , + pos, len, error); +if (mapping && mapping->a_ops->ioerror) mapping->a_ops->ioerror (mapping, direction, pos, len, error);fs/iomap/direct-io.c +5
-0@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)if (dops && dops->end_io) ret = dops->end_io (iocb, dio->size, ret, dio->flags); +if (dio->error) +inode_error (file_inode (iocb->ki_filp), + (dio->flags &IOMAP_DIO_WRITE ) ?FSERR_DIO_WRITE : +FSERR_DIO_READ , + offset, dio->size, dio->error);if (dio->error && dops && dops->ioerror) dops->ioerror(file_inode(iocb->ki_filp), (dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,fs/super.c +53
-0@@ -363,6 +363,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,spin_lock_init (&s->s_inode_list_lock);INIT_LIST_HEAD (&s->s_inodes_wb);spin_lock_init (&s->s_inode_wblist_lock); +BLOCKING_INIT_NOTIFIER_HEAD (&s->s_error_notifier); s->s_count =1 ;atomic_set (&s->s_active,1 );@@ -2267,3 +2268,55 @@ int sb_init_dio_done_wq(struct super_block *sb)return 0 ; }EXPORT_SYMBOL_GPL (sb_init_dio_done_wq); + +static void handle_sb_error (struct work_struct *work) +{ + struct fs_error *fserr = container_of(work, struct fs_error, work); + +fsnotify_sb_error (fserr->sb, fserr->inode, fserr->error); +blocking_notifier_call_chain (&fserr->sb->s_error_notifier, fserr->type, + fserr); +iput (fserr->inode); +kfree (fserr); +} + +/** +* Report a filesystem error. The actual work is deferred to a workqueue so +* that we're always in process context and to avoid blowing out the caller's +* stack. +* +* @sb Filesystem superblock +* @inode Inode within filesystem, if applicable +* @type Type of error +* @pos Start of file range affected, if applicable +* @len Length of file range affected, if applicable +* @error Error encountered. +*/ +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error) +{ +struct fs_error *fserr =kzalloc (sizeof (struct fs_error ),GFP_ATOMIC ); + +if (!fserr) { +printk (KERN_ERR +"lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d" , + inode ? inode->i_ino :0 , type, + pos, len, error); +return ; + } + +if (inode) { + fserr->sb = inode->i_sb; + fserr->inode =igrab (inode); + }else { + fserr->sb = sb; + } + fserr->type = type; + fserr->pos = pos; + fserr->len = len; + fserr->error = error; +INIT_WORK (&fserr->work, handle_sb_error); + +schedule_work (&fserr->work); +} +EXPORT_SYMBOL_GPL (__sb_error);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 102 lines · Jan Kara
Quoted 7 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.
Suggested-by Christoph Hellwig <[email protected]>Signed-off-by "Darrick J. Wong" <[email protected]>Looks good to me. Feel free to add:
Reviewed-by Jan Kara <[email protected]>Honza
Quoted 86 lines
--- include/linux/fs.h | 64 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/iomap/buffered-io.c | 6 +++++ fs/iomap/direct-io.c | 5 ++++ fs/super.c | 53 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 128 insertions(+)include/linux/fs.h +56
-0@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error);
Quoted 8 lines
include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135
Quoted 8 lines
include/linux/fs.h +8
-0+} + +static inline void inode_error (struct inode *inode,enum fs_error_type type, +loff_t pos,u64 len,int error) +{ +__sb_error (inode->i_sb, inode, type, pos, len, error); +} +
Apart from the fact that Christian is not going to be happy with this bloat of fs.h
shouldn't all this be part of fsnotify.h?
I do not see why ext4 should not use the same workqueue or why any code would need to call fsnotify_sb_error() directly.
...
Quoted 95 lines
Quoted 95 lines
include/linux/fs.h +0
-0#endif /* _LINUX_FS_H */ fs/iomap/buffered-io.c +6
-0@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,inline void iomap_mapping_ioerror (struct address_space *mapping,int direction,loff_t pos,u64 len,int error) { +struct inode *inode = mapping->host; + +inode_error (inode, + direction ==READ ?FSERR_READAHEAD :FSERR_WRITEBACK , + pos, len, error); +if (mapping && mapping->a_ops->ioerror) mapping->a_ops->ioerror (mapping, direction, pos, len, error);fs/iomap/direct-io.c +5
-0@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)if (dops && dops->end_io) ret = dops->end_io (iocb, dio->size, ret, dio->flags); +if (dio->error) +inode_error (file_inode (iocb->ki_filp), + (dio->flags &IOMAP_DIO_WRITE ) ?FSERR_DIO_WRITE : +FSERR_DIO_READ , + offset, dio->size, dio->error);if (dio->error && dops && dops->ioerror) dops->ioerror(file_inode(iocb->ki_filp), (dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,fs/super.c +46
-0@@ -363,6 +363,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,spin_lock_init (&s->s_inode_list_lock);INIT_LIST_HEAD (&s->s_inodes_wb);spin_lock_init (&s->s_inode_wblist_lock); +BLOCKING_INIT_NOTIFIER_HEAD (&s->s_error_notifier); s->s_count =1 ;atomic_set (&s->s_active,1 );@@ -2267,3 +2268,55 @@ int sb_init_dio_done_wq(struct super_block *sb)return 0 ; }EXPORT_SYMBOL_GPL (sb_init_dio_done_wq); + +static void handle_sb_error (struct work_struct *work) +{ + struct fs_error *fserr = container_of(work, struct fs_error, work); + +fsnotify_sb_error (fserr->sb, fserr->inode, fserr->error); +blocking_notifier_call_chain (&fserr->sb->s_error_notifier, fserr->type, + fserr); +iput (fserr->inode); +kfree (fserr); +} + +/** +* Report a filesystem error. The actual work is deferred to a workqueue so +* that we're always in process context and to avoid blowing out the caller's +* stack. +* +* @sb Filesystem superblock +* @inode Inode within filesystem, if applicable +* @type Type of error +* @pos Start of file range affected, if applicable +* @len Length of file range affected, if applicable +* @error Error encountered. +*/ +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error) +{ +struct fs_error *fserr =kzalloc (sizeof (struct fs_error ),GFP_ATOMIC ); + +if (!fserr) { +printk (KERN_ERR +"lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d" , + inode ? inode->i_ino :0 , type, + pos, len, error); +return ; + } + +if (inode) { + fserr->sb = inode->i_sb; + fserr->inode =igrab (inode); + }else { + fserr->sb = sb; + } + fserr->type = type; + fserr->pos = pos;
Quoted 8 lines
fs/super.c int sb_init_dio_done_wq(struct super_block *sb) · line 2322
Quoted 8 lines
fs/super.c +7
-0+ fserr->len = len; + fserr->error = error; +INIT_WORK (&fserr->work, handle_sb_error); + +schedule_work (&fserr->work); +} +EXPORT_SYMBOL_GPL (__sb_error);
...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/
Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?
Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?
If this is needed then we could change fsnotify_sb_error() to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.
Thanks,
Amir.
Jan Kara
maintainerRe: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59
-0@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error); +} + +static inline void inode_error(struct inode *inode, enum fs_error_type type,
Quoted 8 lines
Quoted 5 lines
include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135
Quoted 5 lines
include/linux/fs.h +5
-0+ loff_t pos, u64 len, int error) +{ + __sb_error(inode->i_sb, inode, type, pos, len, error); +} +Apart from the fact that Christian is not going to be happy with this bloat of
fs.hshouldn't all this be part offsnotify.h?
Point that this maybe doesn't belong to fs.h is a good one. But I don't think fsnotify.h is appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a new fs_error.h header?
Quoted 2 lines
I do not see why ext4 should not use the same workqueue or why any code would need to call
fsnotify_sb_error()directly.
Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.
Quoted 35 lines
Quoted 21 lines
Quoted 21 lines
include/linux/fs.h +21
-0+void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error) +{ +struct fs_error *fserr =kzalloc (sizeof (struct fs_error ),GFP_ATOMIC ); + +if (!fserr) { +printk (KERN_ERR +"lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d" , + inode ? inode->i_ino :0 , type, + pos, len, error); +return ; + } + +if (inode) { + fserr->sb = inode->i_sb; + fserr->inode =igrab (inode); + }else { + fserr->sb = sb; + } + fserr->type = type; + fserr->pos = pos;Quoted 8 lines
include/linux/fs.h
Quoted 8 lines
include/linux/fs.h +7
-0+ fserr->len = len; + fserr->error = error; +INIT_WORK (&fserr->work, handle_sb_error); + +schedule_work (&fserr->work); +} +EXPORT_SYMBOL_GPL (__sb_error);...
We recently discovered thatfsnotify_sb_error()calls are exposed to races withgeneric_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/
Quoted 2 lines
Will punting all
FS_ERRORevents to workqueue help to improve this situation or will it make it worse?
Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().
Quoted 2 lines
Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?
I think there must be a point of no return during fs shutdown after which we just stop emitting errors.
Quoted 3 lines
If this is needed then we could change
fsnotify_sb_error()to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.
This lifetime issue is not limited to fsnotify. I think __sb_error() needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 92 lines · Jan Kara
Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59
-0@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error); +} + +static inline void inode_error(struct inode *inode, enum fs_error_type type,Quoted 8 lines
Quoted 5 lines
include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135
Quoted 5 lines
include/linux/fs.h +5
-0+ loff_t pos, u64 len, int error) +{ + __sb_error(inode->i_sb, inode, type, pos, len, error); +} +Apart from the fact that Christian is not going to be happy with this bloat of
fs.hshouldn't all this be part offsnotify.h?
Quoted 4 lines
Point that this maybe doesn't belong to
fs.his a good one. But I don't thinkfsnotify.his appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a newfs_error.hheader?
Fine with me, though it'd be a small header.
Quoted 5 lines
Quoted 2 lines
I do not see why ext4 should not use the same workqueue or why any code would need to call
fsnotify_sb_error()directly.Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.
Yes, that should be a trivial patch to change fsnotify_sb_error -> sb_error/inode_error.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
include/linux/fs.h +26
-0+void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error) +{ +struct fs_error *fserr =kzalloc (sizeof (struct fs_error ),GFP_ATOMIC ); + +if (!fserr) { +printk (KERN_ERR +"lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d" , + inode ? inode->i_ino :0 , type, + pos, len, error); +return ; + } + +if (inode) { + fserr->sb = inode->i_sb; + fserr->inode =igrab (inode); + }else { + fserr->sb = sb; + } + fserr->type = type; + fserr->pos = pos; + fserr->len = len; + fserr->error = error; +INIT_WORK (&fserr->work, handle_sb_error); + +schedule_work (&fserr->work);
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
include/linux/fs.h
Quoted 3 lines
include/linux/fs.h +2
-0+} +EXPORT_SYMBOL_GPL (__sb_error);...
We recently discovered thatfsnotify_sb_error()calls are exposed to races withgeneric_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/
Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...
Quoted 7 lines
Quoted 2 lines
Will punting all
FS_ERRORevents to workqueue help to improve this situation or will it make it worse?Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in
handle_sb_error().
I wonder, could __sb_error call get_active_super() to obtain an active reference to the sb, and then deactivate_super() it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.
The igrab/iput was supposed to prevent the same UAF from happening with the inode, but I should've checked for a non-null return value.
Quoted 5 lines
Quoted 2 lines
Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?
I think there must be a point of no return during fs shutdown after which we just stop emitting errors.
I agree, once S_ACTIVE hits zero there's no point in sending further errors.
Quoted 3 lines
Quoted 3 lines
If this is needed then we could change
fsnotify_sb_error()to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.
That would be another way to do it. The sole downstream consumer of the s_error_notifier-based events only cares about ino/gen.
Quoted 4 lines
This lifetime issue is not limited to fsnotify. I think
__sb_error()needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).
<nod>
--D
Quoted 5 lines
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 156 lines · Darrick J. Wong
Quoted 92 lines · Jan Kara
Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59
-0@@ -80,6 +80,7 @@ struct fs_context;struct fs_parameter_spec ;struct file_kattr ;struct iomap_ops ; +struct notifier_head ;extern void __init inode_init (void );extern void __init inode_init_early (void );@@ -1587,6 +1588,7 @@ struct super_block {spinlock_t s_inode_wblist_lock;struct list_head s_inodes_wb;/* writeback inodes */ +struct blocking_notifier_head s_error_notifier; } __randomize_layout;static inline struct user_namespace *i_user_ns (const struct inode *inode)@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,return true; } +enum fs_error_type { +/* pagecache reads and writes */ +FSERR_READAHEAD , +FSERR_WRITEBACK , + +/* directio read and writes */ +FSERR_DIO_READ , +FSERR_DIO_WRITE , + +/* media error */ +FSERR_DATA_LOST , + +/* filesystem metadata */ +FSERR_METADATA , +}; + +struct fs_error { +struct work_struct work; +struct super_block *sb; +struct inode *inode; +loff_t pos; +u64 len; +enum fs_error_type type; +int error; +}; + +struct fs_error_hook { +struct notifier_block nb; +}; + +static inline int sb_hook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +return blocking_notifier_chain_register (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_unhook_error (struct super_block *sb, +struct fs_error_hook *h) +{ +blocking_notifier_chain_unregister (&sb->s_error_notifier, &h->nb); +} + +static inline void sb_init_error_hook (struct fs_error_hook *h,notifier_fn_t fn) +{ + h->nb.notifier_call = fn; + h->nb.priority =0 ; +} + +void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error); + +static inline void sb_error (struct super_block *sb,int error) +{ +__sb_error (sb,NULL ,FSERR_METADATA ,0 ,0 , error); +} + +static inline void inode_error(struct inode *inode, enum fs_error_type type,Quoted 8 lines
Quoted 5 lines
include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135
Quoted 5 lines
include/linux/fs.h +5
-0+ loff_t pos, u64 len, int error) +{ + __sb_error(inode->i_sb, inode, type, pos, len, error); +} +Apart from the fact that Christian is not going to be happy with this bloat of
fs.hshouldn't all this be part offsnotify.h?Quoted 4 lines
Point that this maybe doesn't belong to
fs.his a good one. But I don't thinkfsnotify.his appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a newfs_error.hheader?Fine with me, though it'd be a small header.
Quoted 5 lines
Quoted 2 lines
I do not see why ext4 should not use the same workqueue or why any code would need to call
fsnotify_sb_error()directly.Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.
Yes, that should be a trivial patch to change
fsnotify_sb_error-> sb_error/inode_error.Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
include/linux/fs.h +26
-0+void __sb_error (struct super_block *sb,struct inode *inode, +enum fs_error_type type,loff_t pos,u64 len,int error) +{ +struct fs_error *fserr =kzalloc (sizeof (struct fs_error ),GFP_ATOMIC ); + +if (!fserr) { +printk (KERN_ERR +"lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d" , + inode ? inode->i_ino :0 , type, + pos, len, error); +return ; + } + +if (inode) { + fserr->sb = inode->i_sb; + fserr->inode =igrab (inode); + }else { + fserr->sb = sb; + } + fserr->type = type; + fserr->pos = pos; + fserr->len = len; + fserr->error = error; +INIT_WORK (&fserr->work, handle_sb_error); + +schedule_work (&fserr->work);Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
include/linux/fs.h
Quoted 3 lines
include/linux/fs.h +2
-0+} +EXPORT_SYMBOL_GPL (__sb_error);...
We recently discovered thatfsnotify_sb_error()calls are exposed to races withgeneric_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...
Quoted 7 lines
Quoted 2 lines
Will punting all
FS_ERRORevents to workqueue help to improve this situation or will it make it worse?Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in
handle_sb_error().
Quoted 4 lines
I wonder, could
__sb_errorcallget_active_super()to obtain an active reference to the sb, and thendeactivate_super()it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.
...and now that I've actually tried it, I realize that we can't actually call get_active_super because it can sleep waiting for s_umount and SB_BORN. Maybe we could directly atomic_inc_not_zero(&sb->s_active) adn trust that the caller has an active ref to the sb? I think that's true for anyone calling __sb_error with a non-null inode.
--D
Quoted 34 lines
The igrab/iput was supposed to prevent the same UAF from happening with the inode, but I should've checked for a non-null return value.
Quoted 5 lines
Quoted 2 lines
Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?
I think there must be a point of no return during fs shutdown after which we just stop emitting errors.
I agree, once
S_ACTIVEhits zero there's no point in sending further errors.Quoted 3 lines
Quoted 3 lines
If this is needed then we could change
fsnotify_sb_error()to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.That would be another way to do it. The sole downstream consumer of the s_error_notifier-based events only cares about ino/gen.
Quoted 4 lines
This lifetime issue is not limited to fsnotify. I think
__sb_error()needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).<nod>
--D
Quoted 5 lines
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerRe: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 26 lines · Darrick J. Wong
Quoted 20 lines · Darrick J. Wong
Quoted 6 lines · Jan Kara
Quoted 5 lines · Amir Goldstein
...
We recently discovered thatfsnotify_sb_error()calls are exposed to races withgeneric_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...
Quoted 7 lines
Quoted 2 lines
Will punting all
FS_ERRORevents to workqueue help to improve this situation or will it make it worse?Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in
handle_sb_error().Quoted 4 lines
I wonder, could
__sb_errorcallget_active_super()to obtain an active reference to the sb, and thendeactivate_super()it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.
Quoted 5 lines
...and now that I've actually tried it, I realize that we can't actually call
get_active_superbecause it can sleep waiting fors_umountandSB_BORN. Maybe we could directlyatomic_inc_not_zero(&sb->s_active)adn trust that the caller has an active ref to the sb? I think that's true for anyone calling__sb_errorwith a non-null inode.
Well, the side-effects of holding active sb reference from some workqueue item tend to hit back occasionally (like when userspace assumes the device isn't used anymore but it in fact still is because of the active reference). Every time we tried something like this (last time it was with iouring I believe) some user came back and complained his setup broke. In this case it should be really rare but still I think it's better to avoid it if we can (plus I'm not sure what you'd like to do for __sb_error() callers that don't get the inode and thus active reference isn't really guaranteed - they still need the protection against umount so that handle_sb_error() can do the notifier callchain thing).
So I think a better solution might be that generic_shutdown_super() waits for pending error notifications after clearing SB_ACTIVE before umount proceeds further and __sb_error() just starts discarding new notifications as soon as we see SB_ACTIVE is clear.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 1/6] iomap: report file IO errors to fsnotify
Quoted 44 lines · Jan Kara
Quoted 26 lines · Darrick J. Wong
Quoted 20 lines · Darrick J. Wong
Quoted 6 lines · Jan Kara
Quoted 5 lines · Amir Goldstein
...
We recently discovered thatfsnotify_sb_error()calls are exposed to races withgeneric_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...
Quoted 7 lines
Quoted 2 lines
Will punting all
FS_ERRORevents to workqueue help to improve this situation or will it make it worse?Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in
handle_sb_error().Quoted 4 lines
I wonder, could
__sb_errorcallget_active_super()to obtain an active reference to the sb, and thendeactivate_super()it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.Quoted 5 lines
...and now that I've actually tried it, I realize that we can't actually call
get_active_superbecause it can sleep waiting fors_umountandSB_BORN. Maybe we could directlyatomic_inc_not_zero(&sb->s_active)adn trust that the caller has an active ref to the sb? I think that's true for anyone calling__sb_errorwith a non-null inode.Well, the side-effects of holding active sb reference from some workqueue item tend to hit back occasionally (like when userspace assumes the device isn't used anymore but it in fact still is because of the active reference). Every time we tried something like this (last time it was with iouring I believe) some user came back and complained his setup broke. In this case it should be really rare but still I think it's better to avoid it if we can (plus I'm not sure what you'd like to do for
__sb_error()callers that don't get the inode and thus active reference isn't really guaranteed - they still need the protection against umount so thathandle_sb_error()can do the notifier callchain thing).
Quoted 4 lines
So I think a better solution might be that
generic_shutdown_super()waits for pending error notifications after clearingSB_ACTIVEbefore umount proceeds further and__sb_error()just starts discarding new notifications as soon as we seeSB_ACTIVEis clear.
<nod> Summarizing what we just talked about on the ext4 call--
Instead of grabbing an active reference to the sb, I'll instead fix __sb_error to (a) ignore !BORN || !ACTIVE || DYING supers, and (b) increment a counter in the sb whenever we queue an event, and decrement it when the worker finishes with it. generic_shutdown_super can then wait (having just cleared ACTIVE) for the counter to hit zero before it stops fsnotify and drops the dentry cache.
Or at least that's what I'll try today.
--D
Quoted 5 lines
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 2/6 xfs: switch healthmon to use the iomap I/O error reporting
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use the new generic I/O error reporting paths so that we can remove the xfs-specific hooks.
--- fs/xfs/xfs_trace.h | 35 ++++++++++++++++++--------------- fs/xfs/xfs_healthmon.c | 51 +++++++++++++++++++++++++----------------------- 2 files changed, 46 insertions(+), 40 deletions(-)
fs/xfs/xfs_trace.h +19 -16
@@ -6262,24 +6262,25 @@ TRACE_EVENT(xfs_healthmon_media_error_hook, __entry->lost_prev) ); -#define XFS_FILE_IOERROR_STRINGS \ - { XFS_FILE_IOERROR_BUFFERED_READ, "readahead" }, \ - { XFS_FILE_IOERROR_BUFFERED_WRITE, "writeback" }, \ - { XFS_FILE_IOERROR_DIRECT_READ, "directio_read" }, \ - { XFS_FILE_IOERROR_DIRECT_WRITE, "directio_write" }, \ - { XFS_FILE_IOERROR_DATA_LOST, "datalost" } +#define FS_ERROR_STRINGS \ + { FSERR_READAHEAD, "readahead" }, \ + { FSERR_WRITEBACK, "writeback" }, \ + { FSERR_DIO_READ, "directio_read" }, \ + { FSERR_DIO_WRITE, "directio_write" }, \ + { FSERR_DATA_LOST, "datalost" }, \ + { FSERR_METADATA, "metadata" } - -TRACE_DEFINE_ENUM (XFS_FILE_IOERROR_BUFFERED_READ ); -TRACE_DEFINE_ENUM (XFS_FILE_IOERROR_BUFFERED_WRITE ); -TRACE_DEFINE_ENUM (XFS_FILE_IOERROR_DIRECT_READ ); -TRACE_DEFINE_ENUM (XFS_FILE_IOERROR_DIRECT_WRITE ); -TRACE_DEFINE_ENUM (XFS_FILE_IOERROR_DATA_LOST ); +TRACE_DEFINE_ENUM (FSERR_READAHEAD ); +TRACE_DEFINE_ENUM (FSERR_WRITEBACK ); +TRACE_DEFINE_ENUM (FSERR_DIO_READ ); +TRACE_DEFINE_ENUM (FSERR_DIO_WRITE ); +TRACE_DEFINE_ENUM (FSERR_DATA_LOST ); +TRACE_DEFINE_ENUM (FSERR_METADATA ); TRACE_EVENT(xfs_healthmon_file_ioerror_hook, TP_PROTO(const struct xfs_mount *mp, unsigned long action, - const struct xfs_file_ioerror_params *p, + const struct fs_error *p, unsigned int events, unsigned long long lost_prev), TP_ARGS(mp, action, p, events, lost_prev), TP_STRUCT__entry(
@@ -6294,10 +6295,12 @@ TRACE_EVENT(xfs_healthmon_file_ioerror_hook, __field(unsigned long long, lost_prev) ),TP_fast_assign ( + struct xfs_inode *ip = XFS_I(p->inode); + __entry->dev = mp ? mp->m_super->s_dev :0 ; __entry->action = action; - __entry->ino = p->ino; - __entry->gen = p->gen; + __entry->ino = ip->i_ino; + __entry->gen = p->inode->i_generation; __entry->pos = p->pos; __entry->len = p->len; __entry->events = events;
@@ -6307,7 +6310,7 @@ TRACE_EVENT(xfs_healthmon_file_ioerror_hook,MAJOR (__entry->dev),MINOR (__entry->dev), __entry->ino, __entry->gen, -__print_symbolic (__entry->action,XFS_FILE_IOERROR_STRINGS ), +__print_symbolic (__entry->action,FS_ERROR_STRINGS ), __entry->pos, __entry->len, __entry->events,
fs/xfs/xfs_healthmon.c +27 -24
109 lines not on this page · raw message
PATCH 3/6 xfs: port notify-failure to use the new vfs io error reporting
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port the media error notification code to use the new generic reporting code.
--- fs/xfs/xfs_notify_failure.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
fs/xfs/xfs_notify_failure.c +5 -4
@@ -178,9 +178,10 @@ xfs_dax_failure_fn(invalidate_inode_pages2_range (mapping, pgoff, pgoff + pgcnt -1 ); -xfs_inode_media_error (ip, +inode_error (VFS_I (ip),FSERR_DATA_LOST ,XFS_FSB_TO_B (mp, (u64 )pgoff <<PAGE_SHIFT ), -XFS_FSB_TO_B (mp, (u64 )pgcnt <<PAGE_SHIFT )); +XFS_FSB_TO_B (mp, (u64 )pgcnt <<PAGE_SHIFT ), + -EIO );xfs_irele (ip);return error;
@@ -496,8 +497,8 @@ xfs_report_one_data_lost(if (rmap_end > lost_end) blocks -= rmap_end - lost_end; -xfs_inode_media_error (ip,XFS_FSB_TO_B (mp, fileoff), -XFS_FSB_TO_B (mp, blocks)); +inode_error (VFS_I (ip),FSERR_DATA_LOST ,XFS_FSB_TO_B (mp, fileoff), +XFS_FSB_TO_B (mp, blocks), -EIO );xfs_irele (ip);return 0 ;
PATCH 4/6 xfs: remove file I/O error hooks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove these hooks since iomap now does that on its own.
--- fs/xfs/xfs_file.h | 37 ----------- fs/xfs/xfs_mount.h | 3 - fs/xfs/xfs_aops.c | 2 - fs/xfs/xfs_file.c | 174 ---------------------------------------------------- fs/xfs/xfs_super.c | 1 5 files changed, 1 insertion(+), 216 deletions(-)
fs/xfs/xfs_file.h +0 -37
@@ -12,41 +12,4 @@ extern const struct file_operations xfs_dir_file_operations;bool xfs_is_falloc_aligned (struct xfs_inode *ip,loff_t pos,long long int len); -enum xfs_file_ioerror_type { -XFS_FILE_IOERROR_BUFFERED_READ , -XFS_FILE_IOERROR_BUFFERED_WRITE , -XFS_FILE_IOERROR_DIRECT_READ , -XFS_FILE_IOERROR_DIRECT_WRITE , -XFS_FILE_IOERROR_DATA_LOST , -}; - -struct xfs_file_ioerror_params { -xfs_ino_t ino; -loff_t pos; -u64 len; -u32 gen; -int error; -}; - -#ifdef CONFIG_XFS_LIVE_HOOKS -struct xfs_file_ioerror_hook { -struct xfs_hook ioerror_hook; -}; - -int xfs_file_ioerror_hook_add (struct xfs_mount *mp, -struct xfs_file_ioerror_hook *hook); -void xfs_file_ioerror_hook_del (struct xfs_mount *mp, -struct xfs_file_ioerror_hook *hook); -void xfs_file_ioerror_hook_setup (struct xfs_file_ioerror_hook *hook, -notifier_fn_t mod_fn); - -void xfs_vm_ioerror (struct address_space *mapping,int direction,loff_t pos, -u64 len,int error); - -void xfs_inode_media_error (struct xfs_inode *ip,loff_t pos,u64 len); -#else -# define xfs_vm_ioerror NULL -# define xfs_inode_media_error (...) ((void)0) -#endif /* CONFIG_XFS_LIVE_HOOKS */ -#endif /* __XFS_FILE_H__ */
fs/xfs/xfs_mount.h +0 -3
@@ -352,9 +352,6 @@ typedef struct xfs_mount {/* Hook to feed media error events to a daemon. */ struct xfs_hooks m_media_error_hooks; - -/* Hook to feed file io error events to a daemon. */ -struct xfs_hooks m_file_ioerror_hooks; } xfs_mount_t;#define M_IGEO (mp) (&(mp)->m_ino_geo)
fs/xfs/xfs_aops.c +0 -2
@@ -22,7 +22,6 @@#include "xfs_icache.h" #include "xfs_zone_alloc.h" #include "xfs_rtgroup.h" -#include "xfs_file.h" struct xfs_writepage_ctx {struct iomap_writepage_ctx ctx;
@@ -811,7 +810,6 @@ const struct address_space_operations xfs_address_space_operations = { .is_partially_uptodate = iomap_is_partially_uptodate, .error_remove_folio = generic_error_remove_folio, .swap_activate = xfs_vm_swap_activate, - .ioerror = xfs_vm_ioerror, };const struct address_space_operations xfs_dax_aops = {
fs/xfs/xfs_file.c +1 -173
192 lines not on this page · raw message
fs/xfs/xfs_super.c +0 -1
@@ -2388,7 +2388,6 @@ xfs_init_fs_context(xfs_hooks_init (&mp->m_shutdown_hooks);xfs_hooks_init (&mp->m_health_update_hooks);xfs_hooks_init (&mp->m_media_error_hooks); -xfs_hooks_init (&mp->m_file_ioerror_hooks); fc->s_fs_info = mp; fc->ops = &xfs_context_ops;
PATCH 5/6 iomap: remove I/O error hooks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Remove the I/O error hooks from struct address_space and iomap_dio_ops because there are no more callers.
--- include/linux/fs.h | 4 ---- include/linux/iomap.h | 2 -- Documentation/filesystems/vfs.rst | 7 ------- fs/iomap/buffered-io.c | 4 ---- fs/iomap/direct-io.c | 4 ---- 5 files changed, 21 deletions(-)
include/linux/fs.h +0 -4
@@ -478,10 +478,6 @@ struct address_space_operations { sector_t *span);void (*swap_deactivate)(struct file *file);int (*swap_rw)(struct kiocb *iocb,struct iov_iter *iter); - -/* Callback for dealing with IO errors during readahead or writeback */ -void (*ioerror)(struct address_space *mapping,int direction, -loff_t pos,u64 len,int error); };extern const struct address_space_operations empty_aops;
include/linux/iomap.h +0 -2
@@ -486,8 +486,6 @@ struct iomap_dio_ops { unsigned flags);void (*submit_io)(const struct iomap_iter *iter,struct bio *bio,loff_t file_offset); -void (*ioerror)(struct inode *inode,int direction,loff_t pos, -u64 len,int error);/* * Filesystems wishing to attach private information to a direct io bio
Documentation/filesystems/vfs.rst +0 -7
@@ -822,8 +822,6 @@ cache in your filesystem. The following members are defined: int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); - void (*ioerror)(struct address_space *mapping, int direction, - loff_t pos, u64 len, int error); }; ``read_folio``
@@ -1034,11 +1032,6 @@ cache in your filesystem. The following members are defined: ``swap_rw`` Called to read or write swap pages when SWP_FS_OPS is set. -``ioerror`` - Called to deal with IO errors during readahead or writeback. - This may be called from interrupt context, and without any - locks necessarily being held. - The File Object ===============
fs/iomap/buffered-io.c +0 -4
@@ -296,10 +296,6 @@ inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,inode_error (inode, direction ==READ ?FSERR_READAHEAD :FSERR_WRITEBACK , pos, len, error); - -if (mapping && mapping->a_ops->ioerror) - mapping->a_ops->ioerror (mapping, direction, pos, len, - error); }/**
fs/iomap/direct-io.c +0 -4
@@ -100,10 +100,6 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio) (dio->flags &IOMAP_DIO_WRITE ) ?FSERR_DIO_WRITE :FSERR_DIO_READ , offset, dio->size, dio->error); -if (dio->error && dops && dops->ioerror) - dops->ioerror (file_inode (iocb->ki_filp), - (dio->flags &IOMAP_DIO_WRITE ) ?WRITE :READ , - offset, dio->size, dio->error);if (likely (!ret)) { ret = dio->size;
PATCH 6/6 xfs: report fs metadata errors via fsnotify
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Report filesystem corruption problems to fsnotify.
--- fs/xfs/xfs_health.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
fs/xfs/xfs_health.c +11 -1
@@ -71,6 +71,9 @@ xfs_fs_health_update_hook( unsigned int old_mask, unsigned int new_mask) { +if (op !=XFS_HEALTHUP_HEALTHY && new_mask) +sb_error (mp->m_super, -EFSCORRUPTED ); +if (xfs_hooks_switched_on (&xfs_health_hooks_switch)) {struct xfs_health_update_params p = { .domain = XFS_HEALTHUP_FS,
@@ -91,13 +94,17 @@ xfs_group_health_update_hook(unsigned int old_mask, unsigned int new_mask) { + struct xfs_mount *mp = xg->xg_mount; + +if (op !=XFS_HEALTHUP_HEALTHY && new_mask) +sb_error (mp->m_super, -EFSCORRUPTED ); +if (xfs_hooks_switched_on (&xfs_health_hooks_switch)) {struct xfs_health_update_params p = { .old_mask = old_mask, .new_mask = new_mask, .group = xg->xg_gno, }; -struct xfs_mount *mp = xg->xg_mount;switch (xg->xg_type) {case XG_TYPE_AG :
@@ -124,6 +131,9 @@ xfs_inode_health_update_hook( unsigned int old_mask, unsigned int new_mask) { +if (op !=XFS_HEALTHUP_HEALTHY && new_mask) +inode_error (VFS_I (ip),FSERR_METADATA ,0 ,0 , -EFSCORRUPTED ); +if (xfs_hooks_switched_on (&xfs_health_hooks_switch)) {struct xfs_health_update_params p = { .domain = XFS_HEALTHUP_INODE,