Skip to contentSkip to conversation

xfs: autonomous self healing

PATCH Superseded Darrick J. Wong 5 participants 40 messages Nov 5, 2025 – Nov 27, 2025

Subsystems (6)

v2 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xfs: autonomous self healing

Hi everyone,

You might recall that 18 months ago I showed off an early draft of a patchset implementing autonomous self healing capabilities for XFS. The premise is quite simple -- add a few hooks to the kernel to capture significant filesystem metadata and file health events (pretty much all failures), queue these events to a special anonfd, and let userspace read the events at its leisure. That's patchset 1.

Since the previous release, I've removed all the json event generation stuff and made media errors use the rmap btree to report file data loss. I also ported the userspace program to C. I'm not going to blast everyone with the full set; just know that the C version is here:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring

Patchset 2 is now a cleanup of the file IO error hooks in patchset 1 to use a more generic interface and to call fsnotify with the error reports. This means that the fsnotify filesystem error functionality conveys generic errors to unprivileged userspace programs, but I'm leaving the privileged healthmon interface so that xfsprogs can figure out which specific part of the filesystem needs fixing.

This work was mostly complete by the end of 2024, and I've been letting it run on my XFS QA testing fleets ever since then. I am submitting this patchset for upstream for 6.19. Once this is merged, the online fsck project will be complete.

--D

The unreviewed patches in this series are:

[PATCHSET V3 1/2] xfs: autonomous self healing of filesystems
[PATCH 02/22] docs: discuss autonomous self healing in the xfs online [PATCH 03/22] xfs: create debugfs uuid aliases
[PATCH 04/22] xfs: create hooks for monitoring health updates [PATCH 05/22] xfs: create a filesystem shutdown hook [PATCH 06/22] xfs: create hooks for media errors
[PATCH 07/22] iomap: report buffered read and write io errors to the [PATCH 08/22] iomap: report directio read and write errors to callers [PATCH 09/22] xfs: create file io error hooks
[PATCH 10/22] xfs: create a special file to pass filesystem health to [PATCH 11/22] xfs: create event queuing, formatting, [PATCH 12/22] xfs: report metadata health events through healthmon [PATCH 13/22] xfs: report shutdown events through healthmon [PATCH 14/22] xfs: report media errors through healthmon [PATCH 15/22] xfs: report file io errors through healthmon [PATCH 16/22] xfs: allow reconfiguration of the health monitoring [PATCH 17/22] xfs: validate fds against running healthmon [PATCH 18/22] xfs: add media error reporting ioctl
[PATCH 19/22] xfs: send uevents when major filesystem events happen [PATCH 20/22] xfs: merge health monitoring events when possible [PATCH 21/22] xfs: restrict healthmon users further
[PATCH 22/22] xfs: charge healthmon event objects to the memcg of the
[PATCHSET V3 2/2] iomap: generic file IO error reporting
[PATCH 1/6] iomap: report file IO errors to fsnotify [PATCH 2/6] xfs: switch healthmon to use the iomap I/O error [PATCH 3/6] xfs: port notify-failure to use the new vfs io error [PATCH 4/6] xfs: remove file I/O error hooks
[PATCH 5/6] iomap: remove I/O error hooks
[PATCH 6/6] xfs: report fs metadata errors via fsnotify

#1 lore raw

PATCH v3 1/2 xfs: autonomous self healing of filesystems

Hi all,

This patchset builds new functionality to deliver live information about filesystem health events to userspace. This is done by creating an anonymous file that can be read() for events by userspace programs. Events are captured by hooking various parts of XFS and iomap so that metadata health failures, file I/O errors, and major changes in filesystem state (unmounts, shutdowns, etc.) can be observed by programs.

When an event occurs, the hook functions queue an event object to each event anonfd for later processing. Programs must have CAP_SYS_ADMIN to open the anonfd and there's a maximum event lag to prevent resource overconsumption. The events themselves can be read() from the anonfd either as json objects for human readability, or as C structs for daemons.

In userspace, we create a new daemon program that will read the event objects and initiate repairs automatically. This daemon is managed entirely by systemd and will not block unmounting of the filesystem unless repairs are ongoing. It is autostarted via some udev rules.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
* docs: remove obsolete links in the xfs online repair documentation
* docs: discuss autonomous self healing in the xfs online repair design doc
* xfs: create debugfs uuid aliases
* xfs: create hooks for monitoring health updates
* xfs: create a filesystem shutdown hook
* xfs: create hooks for media errors
* iomap: report buffered read and write io errors to the filesystem
* iomap: report directio read and write errors to callers
* xfs: create file io error hooks
* xfs: create a special file to pass filesystem health to userspace
* xfs: create event queuing, formatting, and discovery infrastructure
* xfs: report metadata health events through healthmon
* xfs: report shutdown events through healthmon
* xfs: report media errors through healthmon
* xfs: report file io errors through healthmon
* xfs: allow reconfiguration of the health monitoring device
* xfs: validate fds against running healthmon
* xfs: add media error reporting ioctl
* xfs: send uevents when major filesystem events happen
* xfs: merge health monitoring events when possible
* xfs: restrict healthmon users further
* xfs: charge healthmon event objects to the memcg of the listening process

---
 fs/iomap/internal.h                                |    2 
 fs/xfs/libxfs/xfs_fs.h                             |  171 ++
 fs/xfs/libxfs/xfs_health.h                         |   52 +
 fs/xfs/xfs_file.h                                  |   37 +
 fs/xfs/xfs_fsops.h                                 |   11 
 fs/xfs/xfs_healthmon.h                             |  108 +
 fs/xfs/xfs_linux.h                                 |    3 
 fs/xfs/xfs_mount.h                                 |   13 
 fs/xfs/xfs_notify_failure.h                        |   39 +
 fs/xfs/xfs_super.h                                 |   13 
 fs/xfs/xfs_trace.h                                 |  407 ++++++
 include/linux/fs.h                                 |    4 
 include/linux/iomap.h                              |    2 
 Documentation/filesystems/vfs.rst                  |    7 
 .../filesystems/xfs/xfs-online-fsck-design.rst     |  450 +++---
 fs/iomap/buffered-io.c                             |   27 
 fs/iomap/direct-io.c                               |    4 
 fs/iomap/ioend.c                                   |    4 
 fs/xfs/Kconfig                                     |    8 
 fs/xfs/Makefile                                    |    7 
 fs/xfs/xfs_aops.c                                  |    2 
 fs/xfs/xfs_file.c                                  |  174 ++
 fs/xfs/xfs_fsops.c                                 |   60 +
 fs/xfs/xfs_health.c                                |  271 ++++
 fs/xfs/xfs_healthmon.c                             | 1466 ++++++++++++++++++++
 fs/xfs/xfs_ioctl.c                                 |    7 
 fs/xfs/xfs_notify_failure.c                        |  258 +++-
 fs/xfs/xfs_super.c                                 |  109 +
 fs/xfs/xfs_trace.c                                 |    4 
 lib/seq_buf.c                                      |    1 
 30 files changed, 3477 insertions(+), 244 deletions(-)
 create mode 100644 fs/xfs/xfs_healthmon.h
 create mode 100644 fs/xfs/xfs_healthmon.c
#2 lore raw

PATCH 1/22 docs: remove obsolete links in the xfs online repair documentation

From: Darrick J. Wong <[email protected]>

Online repair is now merged in upstream, no need to point to patchset links anymore.

---
 .../filesystems/xfs/xfs-online-fsck-design.rst     |  236 +-------------------
 1 file changed, 6 insertions(+), 230 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +6 -230
index 8cbcd3c2643430..189d1f5f40788d 100644
@@ -105,10 +105,8 @@ occur; this capability aids both strategies.
 TLDR; Show Me the Code!
 -----------------------
 
-Code is posted to the kernel.org git trees as follows:
-`kernel changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-symlink>`_,
-`userspace changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_, and
-`QA test changes <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs>`_.
+Kernel and userspace code has been fully merged as of October 2025.
+
 Each kernel patchset adding an online repair function will use the same branch
 name across the kernel, xfsprogs, and fstests git repos.
 
@@ -764,12 +762,8 @@ allow the online fsck developers to compare online fsck against offline fsck,
 and they enable XFS developers to find deficiencies in the code base.
 
 Proposed patchsets include
-`general fuzzer improvements
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_,
 `fuzzing baselines
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_,
-and `improvements in fuzz testing comprehensiveness
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_.
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_.
 
 Stress Testing
 --------------
@@ -801,11 +795,6 @@ Success is defined by the ability to run all of these tests without observing
 any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
 check warnings, or any other sort of mischief.
 
-Proposed patchsets include `general stress testing
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_
-and the `evolution of existing per-function stress testing
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_.
-
 4. User Interface
 =================
 
@@ -886,10 +875,6 @@ apply as nice of a priority to IO and CPU scheduling as possible.
 This measure was taken to minimize delays in the rest of the filesystem.
 No such hardening has been performed for the cron job.
 
-Proposed patchset:
-`Enabling the xfs_scrub background service
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_.
-
 Health Reporting
 ----------------
 
@@ -912,13 +897,6 @@ notifications and initiate a repair?
 *Answer*: These questions remain unanswered, but should be a part of the
 conversation with early adopters and potential downstream users of XFS.
 
-Proposed patchsets include
-`wiring up health reports to correction returns
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_
-and
-`preservation of sickness info during memory reclaim
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_.
-
 5. Kernel Algorithms and Data Structures
 ========================================
 
@@ -1310,21 +1288,6 @@ Space allocation records are cross-referenced as follows:
      are there the same number of reverse mapping records for each block as the
      reference count record claims?
 
-Proposed patchsets are the series to find gaps in
-`refcount btree
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_,
-`inode btree
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and
-`rmap btree
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records;
-to find
-`mergeable records
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_;
-and to
-`improve cross referencing with rmap
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_
-before starting a repair.
-
 Checking Extended Attributes
 ````````````````````````````
 
@@ -1756,10 +1719,6 @@ For scrub, the drain works as follows:
 To avoid polling in step 4, the drain provides a waitqueue for scrub threads to
 be woken up whenever the intent count drops to zero.
 
-The proposed patchset is the
-`scrub intent drain series
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_.
-
 .. _jump_labels:
 
 Static Keys (aka Jump Label Patching)
@@ -2036,10 +1995,6 @@ The ``xfarray_store_anywhere`` function is used to insert a record in any
 null record slot in the bag; and the ``xfarray_unset`` function removes a
 record from the bag.
 
-The proposed patchset is the
-`big in-memory array
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array>`_.
-
 Iterating Array Elements
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -2172,10 +2127,6 @@ However, it should be noted that these repair functions only use blob storage
 to cache a small number of entries before adding them to a temporary ondisk
 file, which is why compaction is not required.
 
-The proposed patchset is at the start of the
-`extended attribute repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_ series.
-
 .. _xfbtree:
 
 In-Memory B+Trees
@@ -2214,11 +2165,6 @@ xfiles enables reuse of the entire btree library.
 Btrees built atop an xfile are collectively known as ``xfbtrees``.
 The next few sections describe how they actually work.
 
-The proposed patchset is the
-`in-memory btree
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=in-memory-btrees>`_
-series.
-
 Using xfiles as a Buffer Cache Target
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -2459,14 +2405,6 @@ This enables the log to release the old EFI to keep the log moving forwards.
 EFIs have a role to play during the commit and reaping phases; please see the
 next section and the section about :ref:`reaping<reaping>` for more details.
 
-Proposed patchsets are the
-`bitmap rework
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework>`_
-and the
-`preparation for bulk loading btrees
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_.
-
-
 Writing the New Tree
 ````````````````````
 
@@ -2623,11 +2561,6 @@ The number of records for the inode btree is the number of xfarray records,
 but the record count for the free inode btree has to be computed as inode chunk
 records are stored in the xfarray.
 
-The proposed patchset is the
-`AG btree repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
-series.
-
 Case Study: Rebuilding the Space Reference Counts
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -2716,11 +2649,6 @@ Reverse mappings are added to the bag using ``xfarray_store_anywhere`` and
 removed via ``xfarray_unset``.
 Bag members are examined through ``xfarray_iter`` loops.
 
-The proposed patchset is the
-`AG btree repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
-series.
-
 Case Study: Rebuilding File Fork Mapping Indices
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -2757,11 +2685,6 @@ EXTENTS format instead of BMBT, which may require a conversion.
 Third, the incore extent map must be reloaded carefully to avoid disturbing
 any delayed allocation extents.
 
-The proposed patchset is the
-`file mapping repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings>`_
-series.
-
 .. _reaping:
 
 Reaping Old Metadata Blocks
@@ -2843,11 +2766,6 @@ blocks.
 As stated earlier, online repair functions use very large transactions to
 minimize the chances of this occurring.
 
-The proposed patchset is the
-`preparation for bulk loading btrees
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading>`_
-series.
-
 Case Study: Reaping After a Regular Btree Repair
 ````````````````````````````````````````````````
 
@@ -2943,11 +2861,6 @@ When the walk is complete, the bitmap disunion operation ``(ag_owner_bitmap &
 btrees.
 These blocks can then be reaped using the methods outlined above.
 
-The proposed patchset is the
-`AG btree repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
-series.
-
 .. _rmap_reap:
 
 Case Study: Reaping After Repairing Reverse Mapping Btrees
@@ -2972,11 +2885,6 @@ methods outlined above.
 The rest of the process of rebuildng the reverse mapping btree is discussed
 in a separate :ref:`case study<rmap_repair>`.
 
-The proposed patchset is the
-`AG btree repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees>`_
-series.
-
 Case Study: Rebuilding the AGFL
 ```````````````````````````````
 
@@ -3024,11 +2932,6 @@ more complicated, because computing the correct value requires traversing the
 forks, or if that fails, leaving the fields invalid and waiting for the fork
 fsck functions to run.
 
-The proposed patchset is the
-`inode
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes>`_
-repair series.
-
 Quota Record Repairs
 --------------------
 
@@ -3045,11 +2948,6 @@ checking are obviously bad limits and timer values.
 Quota usage counters are checked, repaired, and discussed separately in the
 section about :ref:`live quotacheck <quotacheck>`.
 
-The proposed patchset is the
-`quota
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota>`_
-repair series.
-
 .. _fscounters:
 
 Freezing to Fix Summary Counters
@@ -3145,11 +3043,6 @@ long enough to check and correct the summary counters.
 |   This bug was fixed in Linux 5.17.                                      |
 +--------------------------------------------------------------------------+
 
-The proposed patchset is the
-`summary counter cleanup
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters>`_
-series.
-
 Full Filesystem Scans
 ---------------------
 
@@ -3277,15 +3170,6 @@ Second, if the incore inode is stuck in some intermediate state, the scan
 coordinator must release the AGI and push the main filesystem to get the inode
 back into a loadable state.
 
-The proposed patches are the
-`inode scanner
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
-series.
-The first user of the new functionality is the
-`online quotacheck
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
-series.
-
 Inode Management
 ````````````````
 
@@ -3381,12 +3265,6 @@ To capture these nuances, the online fsck code has a separate ``xchk_irele``
 function to set or clear the ``DONTCACHE`` flag to get the required release
 behavior.
 
-Proposed patchsets include fixing
-`scrub iget usage
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iget-fixes>`_ and
-`dir iget usage
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-dir-iget-fixes>`_.
-
 .. _ilocking:
 
 Locking Inodes
@@ -3443,11 +3321,6 @@ If the dotdot entry changes while the directory is unlocked, then a move or
 rename operation must have changed the child's parentage, and the scan can
 exit early.
 
-The proposed patchset is the
-`directory repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
-series.
-
 .. _fshooks:
 
 Filesystem Hooks
@@ -3594,11 +3467,6 @@ The inode scan APIs are pretty simple:
 
 - ``xchk_iscan_teardown`` to finish the scan
 
-This functionality is also a part of the
-`inode scanner
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-iscan>`_
-series.
-
 .. _quotacheck:
 
 Case Study: Quota Counter Checking
@@ -3686,11 +3554,6 @@ needing to hold any locks for a long duration.
 If repairs are desired, the real and shadow dquots are locked and their
 resource counts are set to the values in the shadow dquot.
 
-The proposed patchset is the
-`online quotacheck
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck>`_
-series.
-
 .. _nlinks:
 
 Case Study: File Link Count Checking
@@ -3744,11 +3607,6 @@ shadow information.
 If no parents are found, the file must be :ref:`reparented <orphanage>` to the
 orphanage to prevent the file from being lost forever.
 
-The proposed patchset is the
-`file link count repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks>`_
-series.
-
 .. _rmap_repair:
 
 Case Study: Rebuilding Reverse Mapping Records
@@ -3828,11 +3686,6 @@ scan for reverse mapping records.
 
 12. Free the xfbtree now that it not needed.
 
-The proposed patchset is the
-`rmap repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree>`_
-series.
-
 Staging Repairs with Temporary Files on Disk
 --------------------------------------------
 
@@ -3971,11 +3824,6 @@ Once a good copy of a data file has been constructed in a temporary file, it
 must be conveyed to the file being repaired, which is the topic of the next
 section.
 
-The proposed patches are in the
-`repair temporary files
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-tempfiles>`_
-series.
-
 Logged File Content Exchanges
 -----------------------------
 
@@ -4025,11 +3873,6 @@ The new ``XFS_SB_FEAT_INCOMPAT_EXCHRANGE`` incompatible feature flag
 in the superblock protects these new log item records from being replayed on
 old kernels.
 
-The proposed patchset is the
-`file contents exchange
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates>`_
-series.
-
 +--------------------------------------------------------------------------+
 | **Sidebar: Using Log-Incompatible Feature Flags**                        |
 +--------------------------------------------------------------------------+
@@ -4323,11 +4166,6 @@ To repair the summary file, write the xfile contents into the temporary file
 and use atomic mapping exchange to commit the new contents.
 The temporary file is then reaped.
 
-The proposed patchset is the
-`realtime summary repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary>`_
-series.
-
 Case Study: Salvaging Extended Attributes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -4369,11 +4207,6 @@ Salvaging extended attributes is done as follows:
 
 4. Reap the temporary file.
 
-The proposed patchset is the
-`extended attribute repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs>`_
-series.
-
 Fixing Directories
 ------------------
 
@@ -4448,11 +4281,6 @@ Unfortunately, the current dentry cache design doesn't provide a means to walk
 every child dentry of a specific directory, which makes this a hard problem.
 There is no known solution.
 
-The proposed patchset is the
-`directory repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs>`_
-series.
-
 Parent Pointers
 ```````````````
 
@@ -4612,11 +4440,6 @@ a :ref:`directory entry live update hook <liveupdate>` as follows:
 
 7. Reap the temporary directory.
 
-The proposed patchset is the
-`parent pointers directory repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
-series.
-
 Case Study: Repairing Parent Pointers
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -4662,11 +4485,6 @@ directory reconstruction:
 
 8. Reap the temporary file.
 
-The proposed patchset is the
-`parent pointers repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-fsck>`_
-series.
-
 Digression: Offline Checking of Parent Pointers
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -4755,11 +4573,6 @@ connectivity checks:
 
 4. Move on to examining link counts, as we do today.
 
-The proposed patchset is the
-`offline parent pointers repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fsck>`_
-series.
-
 Rebuilding directories from parent pointers in offline repair would be very
 challenging because xfs_repair currently uses two single-pass scans of the
 filesystem during phases 3 and 4 to decide which files are corrupt enough to be
@@ -4903,12 +4716,6 @@ Repairing the directory tree works as follows:
 
 6. If the subdirectory has zero paths, attach it to the lost and found.
 
-The proposed patches are in the
-`directory tree repair
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-directory-tree>`_
-series.
-
-
 .. _orphanage:
 
 The Orphanage
@@ -4973,11 +4780,6 @@ Orphaned files are adopted by the orphanage as follows:
 7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all
    resources.
 
-The proposed patches are in the
-`orphanage adoption
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-orphanage>`_
-series.
-
 6. Userspace Algorithms and Data Structures
 ===========================================
 
@@ -5091,14 +4893,6 @@ first workqueue's workers until the backlog eases.
 This doesn't completely solve the balancing problem, but reduces it enough to
 move on to more pressing issues.
 
-The proposed patchsets are the scrub
-`performance tweaks
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-performance-tweaks>`_
-and the
-`inode scan rebalance
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance>`_
-series.
-
 .. _scrubrepair:
 
 Scheduling Repairs
@@ -5179,20 +4973,6 @@ immediately.
 Corrupt file data blocks reported by phase 6 cannot be recovered by the
 filesystem.
 
-The proposed patchsets are the
-`repair warning improvements
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-better-repair-warnings>`_,
-refactoring of the
-`repair data dependency
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps>`_
-and
-`object tracking
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-object-tracking>`_,
-and the
-`repair scheduling
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-scheduling>`_
-improvement series.
-
 Checking Names for Confusable Unicode Sequences
 -----------------------------------------------
 
@@ -5372,6 +5152,8 @@ The extra flexibility enables several new use cases:
   This emulates an atomic device write in software, and can support arbitrary
   scattered writes.
 
+(This functionality was merged into mainline as of 2025)
+
 Vectorized Scrub
 ----------------
 
@@ -5393,13 +5175,7 @@ It is hoped that ``io_uring`` will pick up enough of this functionality that
 online fsck can use that instead of adding a separate vectored scrub system
 call to XFS.
 
-The relevant patchsets are the
-`kernel vectorized scrub
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=vectorized-scrub>`_
-and
-`userspace vectorized scrub
-<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=vectorized-scrub>`_
-series.
+(This functionality was merged into mainline as of 2025)
 
 Quality of Service Targets for Scrub
 ------------------------------------
#3 lore raw

Re: [PATCH 01/22] docs: remove obsolete links in the xfs online repair documentation

Quoted 7 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Online repair is now merged in upstream, no need to point to patchset links anymore.

Can we get this queued up for 6.19? Sending it standalone might be best, although given that Carlos is on Cc maybe he can directly pick it up?

maintainer
#4 lore raw

Re: [PATCH 01/22] docs: remove obsolete links in the xfs online repair documentation

Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Online repair is now merged in upstream, no need to point to patchset links anymore.

Quoted 7 lines
Quoted 2 lines

Can we get this queued up for 6.19? Sending it standalone might be best, although given that Carlos is on Cc maybe he can directly pick it up?

Done, will send an update soon.

#5 lore raw

PATCH 2/22 docs: discuss autonomous self healing in the xfs online repair design doc

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Update the XFS online repair document to describe the motivation and design of the autonomous filesystem healing agent known as xfs_healer.

---
 .../filesystems/xfs/xfs-online-fsck-design.rst     |  218 ++++++++++++++++++++
 1 file changed, 216 insertions(+), 2 deletions(-)
Documentation/filesystems/xfs/xfs-online-fsck-design.rst +216 -2
index 189d1f5f40788d..79d5aa78f2a8bf 100644
@@ -166,9 +166,12 @@ The current XFS tools leave several problems unsolved:
    malicious actors **exploit quirks of Unicode** to place misleading names
    in directories.
 
+8. **Site Reliability and Support Engineers** would like to reduce the
+   frequency of incidents requiring **manual intervention**.
+
 Given this definition of the problems to be solved and the actors who would
 benefit, the proposed solution is a third fsck tool that acts on a running
-filesystem.
+filesystem, and an autononmous agent that fixes problems as they arise.
 
 This new third program has three components: an in-kernel facility to check
 metadata, an in-kernel facility to repair metadata, and a userspace driver
@@ -203,6 +206,13 @@ Even if a piece of filesystem metadata can only be regenerated by scanning the
 entire system, the scan can still be done in the background while other file
 operations continue.
 
+The autonomous self healing agent should listen for metadata health impact
+reports coming from the kernel and automatically schedule repairs for the
+damaged metadata.
+If the required repairs are larger in scope than a single metadata structure,
+``xfs_scrub`` should be invoked to perform a full analysis.
+``xfs_healer`` is the name of this program.
+
 In summary, online fsck takes advantage of resource sharding and redundant
 metadata to enable targeted checking and repair operations while the system
 is running.
@@ -850,11 +860,16 @@ variable in the following service files:
 * ``xfs_scrub_all_fail.service``
 
 The decision to enable the background scan is left to the system administrator.
-This can be done by enabling either of the following services:
+This can be done system-wide by enabling either of the following services:
 
 * ``xfs_scrub_all.timer`` on systemd systems
 * ``xfs_scrub_all.cron`` on non-systemd systems
 
+To enable online repair for specific filesystems, the ``autofsck``
+filesystem property should be set to ``repair``.
+To enable only scanning, the property should be set to ``check``.
+To disable online fsck entirely, the property should be set to ``none``.
+
 This automatic weekly scan is configured out of the box to perform an
 additional media scan of all file data once per month.
 This is less foolproof than, say, storing file data block checksums, but much
@@ -897,6 +912,36 @@ notifications and initiate a repair?
 *Answer*: These questions remain unanswered, but should be a part of the
 conversation with early adopters and potential downstream users of XFS.
 
+Autonomous Self Healing
+-----------------------
+
+The autonomous self healing agent is a background system service that starts
+when the filesystem is mounted and runs until unmount.
+When starting up, the agent opens a special pseudofile under the specific
+mount.
+When the filesystem generates new adverse health events, the events will be
+made available for reading via the special pseudofile.
+The events need not be limited to metadata concerns; they can also reflect
+events outside of the filesystem's direct control such as file I/O errors.
+
+The agent reads these events in a loop and responds to the events
+appropriately.
+For a single trouble report about metadata, the agent initiates a targeted
+repair of the specific structure.
+If that repair fails or the agent observes too many metadata trouble reports
+over a short interval, it should then initiate a full scan of the filesystem
+via the ``xfs_scrub`` service.
+
+The decision to enable the background scan is left to the system administrator.
+This can be done system-wide by enabling the following services:
+
+* ``[email protected]`` on systemd systems
+
+To enable autonomous healing for specific filesystems, the ``autofsck``
+filesystem property should be set to ``repair``.
+To disable self healing, the property should be set to ``check``,
+``optimize``, or ``none``.
+
 5. Kernel Algorithms and Data Structures
 ========================================
 
@@ -4780,6 +4825,136 @@ Orphaned files are adopted by the orphanage as follows:
 7. If a runtime error happens, call ``xrep_adoption_cancel`` to release all
    resources.
 
+Health Monitoring
+-----------------
+
+A self-correcting filesystem responds to observations of problems by scheduling
+repairs of the affected areas.
+The filesystem must therefore create event objects in response to stimuli
+(metadata corruption, file I/O errors, etc.) and dispatch these events to
+downstream consumers.
+Downstream consumers that are in the kernel itself are easy to implement with
+the ``xfs_hooks`` infrastructure created for other parts of online repair; these
+are basically indirect function calls.
+
+However, the decision to translate an adverse metadata health report into a
+repair should be made by userspace, and the actual scheduling done by userspace.
+Some users (e.g. containers) would prefer to fast-fail the container and restart
+it on another node at a previous checkpoint.
+For workloads running in isolation, repairs may be preferable; either way this
+is something the system administrator knows, and not the kernel.
+A userspace agent (``xfs_healer``, described later) will collect events from the
+kernel and dispatch them appropriately.
+
+Exporting health events to userspace requires the creation of a new component,
+known as the health monitor.
+Because the monitor exposes itself to userspace to deliver information, a file
+descriptor is the natural abstraction to use here.
+The health monitor hooks all the relevant sources of metadata health events.
+Upon activation of the hook, a new event object is created and added to a queue.
+When the agent reads from the fd, event objects are pulled from the start of the
+queue and formatted into the user's buffer.
+The events are freed, and the read call returns to userspace to allow the agent
+to perform some work.
+Memory usage is constrained on a per-fd basis to prevent memory exhaustion; if
+an event must be discarded, a special "lost event" event is delivered to the
+agent.
+
+In short, health events are captured, queued, and eventually copied out to
+userspace for dispatching.
+
+**Question**: Why use a pseudofile and not use existing notification methods?
+
+*Answer*: The pseudofile is a private filesystem interface only available to
+processes with the CAP_SYS_ADMIN priviledge and the ability to open the root
+directory.
+Being private gives the kernel and ``xfs_healer`` the flexibility to change
+or update the event format in the future without worrying about backwards
+compatibility.
+Using existing notifications means that the event format would be frozen in
+the public fsnotify UAPI forever, which would affect two subsystems.
+
+The pseudofile can also accept ioctls, which gives ``xfs_healer`` a solid
+means to validate that prior to a repair, its reopened mountpoint is actually
+the same filesystem that is being monitored.
+
+**Question**: Why not reuse fs/notify?
+
+*Answer*: It's much simpler for the healthmon code to manage its own queue of
+events and to wake up readers instead of reusing fsnotify because that's the
+only part of fsnotify that would use.
+
+Before I get started, an introduction: fsnotify expects its users (e.g.
+fanotify) to implement quite a bit of functionality; all it provides is a
+wrapper around a simple queue and a lot of code to convey information about the
+calling process to that user.
+fanotify has to actually implement all the queue management code on its own,
+and so would healthmon.
+
+So if healthmon used fsnotify, it would have to create its own fsnotify group
+structure.
+For our purposes, the group is a very large wrapper around a linked list, some
+counters, and a mutex.
+The group object is critical for ensuring that sees only its own events, and
+that nobody else (e.g. regular fanotify) ever sees these events.
+There's a lot more in there for controlling whether fanotify reports pids,
+groups, file handles, etc. that healthmon doesn't care about.
+
+Starting from the fsnotify() function call:
+
+ - I /think/ we'd have to define a new "data type", which itself is just a plain
+   int but I think they correspond to FSNOTIFY_EVENT_* values which themselves
+   are actually part of an enum.
+   The data type controls the typecasting options for the ``void *data``
+   parameter, which I guess is how I'd pass the healthmon event info from the
+   hooks into the fsnotify mechanism and back out to the healthmon code.
+
+ - Each filesystem that wants to do this probably has to add their own
+   FSNOTIFY_EVENT_{XFS,BTRFS,BFS} data type value because that's a casting
+   decision that's made inside the main fsnotify code.
+   I think this can be avoided if each fs is careful never to leak events
+   outside of the group.
+   Either way, it's harder to follow the data flows here because fsnotify can
+   only take and pass around ``void *`` pointers, and it makes various indirect
+   function calls to manage events.
+   Contrast this with doing everything with typed pointers and direct calls
+   within ``xfs_healthmon.c``.
+
+ - Since healthmon is both producer and consumer of fsnotify events, we can
+   probably define our own "mask" value.
+   It's a relief that we don't have to interact with fanotify, because fanotify
+   has used up 22 of its 32 mask bits.
+
+Once healthmon gets an event into fsnotify, fsnotify will call back (into
+healthmon!) to tell it that it got an event.
+From there, the fsnotify implementation (healthmon) has to allocate an event
+object and add it to the event queue in the group, which is what it already does
+now.
+Overflow control is up to the fsnotify implementation, which healthmon already
+implements.
+
+After the event is queued, the fsnotify implementation also has to implement its
+own read file op to dequeue an event and copy it to the userspace buffer in
+whatever format it likes.
+Again, healthmon already does all this.
+
+In the end, replacing the homegrown event dispatching in healthmon with fsnotify
+would make the data flows much harder to understand, and all we gain is a
+generic event dispatcher that relies on indirect function calls instead of
+direct ones.
+We still have to implement the queuing discipline ourselves! :(
+
+**Future Work Question**: Should these events be exposed through the fanotify
+filesystem error event interface?
+
+*Answer*: Yes.
+fanotify is much more careful about filtering out events to processes that
+aren't running with privileges.
+These processes should have a means to receive simple notifications about
+file errors.
+However, this will require coordination between fanotify, ext4, and XFS, and
+is (for now) outside the scope of this project.
+
 6. Userspace Algorithms and Data Structures
 ===========================================
 
@@ -5071,6 +5246,45 @@ and report what has been lost.
 For media errors in blocks owned by files, parent pointers can be used to
 construct file paths from inode numbers for user-friendly reporting.
 
+Autonomous Self Healing
+-----------------------
+
+When a filesystem mounts, the Linux kernel initiates a uevent describing the
+mount and the path to the data device.
+A udev rule determines the initial mountpoint from the data device path
+and starts a mount-specific ``xfs_healer`` service instance.
+The ``xfs_healer`` service opens the mountpoint and issues the
+XFS_IOC_HEALTH_MONITOR ioctl to open a special health monitoring file.
+After that is set up, the mountpoint is closed to avoid pinning the mount.
+
+The health monitoring file hooks certain points of the filesystem so that it
+may receive events about metadata health, filesystem shutdowns, media errors,
+file I/O errors, and unmounting of the filesystem.
+Events are queued up for each health monitor file and encoded into a
+``struct xfs_health_monitor_event`` object when the agent calls ``read()`` on
+the file.
+All health events are dispatched to a background threadpool to reduce stalls
+in the main event loop.
+Events can be logged into the system log for further analysis.
+
+For metadata health events, the specific details are used to construct a call
+to the scrub ioctl.
+The filesystem mountpoint is reopened, and the kernel is called.
+If events are lost or the repairs fail, a full scan will be initiated by
+starting up an ``[email protected]`` for the given mountpoint.
+
+A filesystem shutdown causes all future repair work to cease, and an unmount
+causes the agent to exit.
+
+**Future Work Question**: Should the healer daemon also register a dbus
+listener and publish events there?
+
+*Answer*: This is unclear -- if there's a demand for system monitoring daemons
+to consume this information and make decisions, then yes, this could be wired
+up in ``xfs_healer``.
+On the other hand, systemd is in the middle of a transition to varlink, so
+it makes more sense to wait and see what happens.
+
 7. Conclusion and Future Work
 =============================
 
#6 lore raw

PATCH 3/22 xfs: create debugfs uuid aliases

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create an alias for the debugfs dir so that we can find a filesystem by uuid. Unless it's mounted nouuid.

---
 fs/xfs/xfs_mount.h |    1 +
 fs/xfs/xfs_super.c |   11 +++++++++++
 2 files changed, 12 insertions(+)
fs/xfs/xfs_mount.h +1 -0
index b871dfde372b52..94108668ddabbd 100644
@@ -289,6 +289,7 @@ typedef struct xfs_mount {
 	struct delayed_work	m_reclaim_work;	/* background inode reclaim */
 	struct xfs_zone_info	*m_zone_info;	/* zone allocator information */
 	struct dentry		*m_debugfs;	/* debugfs parent */
+	struct dentry		*m_debugfs_uuid; /* debugfs symlink */
 	struct xfs_kobj		m_kobj;
 	struct xfs_kobj		m_error_kobj;
 	struct xfs_kobj		m_error_meta_kobj;
fs/xfs/xfs_super.c +11 -0
index 1067ebb3b001bf..ba07e4a4ae3ffa 100644
@@ -819,6 +819,7 @@ xfs_mount_free(
 	if (mp->m_ddev_targp)
 		xfs_free_buftarg(mp->m_ddev_targp);
 
+	debugfs_remove(mp->m_debugfs_uuid);
 	debugfs_remove(mp->m_debugfs);
 	kfree(mp->m_rtname);
 	kfree(mp->m_logname);
@@ -1969,6 +1970,16 @@ xfs_fs_fill_super(
 		goto out_unmount;
 	}
 
+	if (xfs_debugfs && mp->m_debugfs && !xfs_has_nouuid(mp)) {
+		char	name[UUID_STRING_LEN + 1];
+
+		snprintf(name, UUID_STRING_LEN + 1, "%pU", &mp->m_sb.sb_uuid);
+		mp->m_debugfs_uuid = debugfs_create_symlink(name, xfs_debugfs,
+				mp->m_super->s_id);
+	} else {
+		mp->m_debugfs_uuid = NULL;
+	}
+
 	return 0;
 
  out_filestream_unmount:
#7 lore raw

PATCH 4/22 xfs: create hooks for monitoring health updates

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create hooks for monitoring health events.

---
 fs/xfs/libxfs/xfs_health.h |   47 ++++++++++
 fs/xfs/xfs_mount.h         |    3 +
 fs/xfs/xfs_health.c        |  204 ++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_super.c         |    1 
 4 files changed, 254 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_health.h +47 -0
index b31000f7190ce5..39fef33dedc6a8 100644
@@ -289,4 +289,51 @@ void xfs_bulkstat_health(struct xfs_inode *ip, struct xfs_bulkstat *bs);
 #define xfs_metadata_is_sick(error) \
 	(unlikely((error) == -EFSCORRUPTED || (error) == -EFSBADCRC))
 
+/*
+ * Parameters for tracking health updates.  The enum below is passed as the
+ * hook function argument.
+ */
+enum xfs_health_update_type {
+	XFS_HEALTHUP_SICK = 1,	/* runtime corruption observed */
+	XFS_HEALTHUP_CORRUPT,	/* fsck reported corruption */
+	XFS_HEALTHUP_HEALTHY,	/* fsck reported healthy structure */
+	XFS_HEALTHUP_UNMOUNT,	/* filesystem is unmounting */
+};
+
+/* Where in the filesystem was the event observed? */
+enum xfs_health_update_domain {
+	XFS_HEALTHUP_FS = 1,	/* main filesystem */
+	XFS_HEALTHUP_AG,	/* allocation group */
+	XFS_HEALTHUP_INODE,	/* inode */
+	XFS_HEALTHUP_RTGROUP,	/* realtime group */
+};
+
+struct xfs_health_update_params {
+	/* XFS_HEALTHUP_INODE */
+	xfs_ino_t			ino;
+	uint32_t			gen;
+
+	/* XFS_HEALTHUP_AG/RTGROUP */
+	uint32_t			group;
+
+	/* XFS_SICK_* flags */
+	unsigned int			old_mask;
+	unsigned int			new_mask;
+
+	enum xfs_health_update_domain	domain;
+};
+
+#ifdef CONFIG_XFS_LIVE_HOOKS
+struct xfs_health_hook {
+	struct xfs_hook			health_hook;
+};
+
+void xfs_health_hook_disable(void);
+void xfs_health_hook_enable(void);
+
+int xfs_health_hook_add(struct xfs_mount *mp, struct xfs_health_hook *hook);
+void xfs_health_hook_del(struct xfs_mount *mp, struct xfs_health_hook *hook);
+void xfs_health_hook_setup(struct xfs_health_hook *hook, notifier_fn_t mod_fn);
+#endif /* CONFIG_XFS_LIVE_HOOKS */
+
 #endif	/* __XFS_HEALTH_H__ */
fs/xfs/xfs_mount.h +3 -0
index 94108668ddabbd..3f20baaf9cc226 100644
@@ -343,6 +343,9 @@ typedef struct xfs_mount {
 
 	/* Hook to feed dirent updates to an active online repair. */
 	struct xfs_hooks	m_dir_update_hooks;
+
+	/* Hook to feed health events to a daemon. */
+	struct xfs_hooks	m_health_update_hooks;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_health.c +203 -1

350 lines not on this page · raw message

fs/xfs/xfs_super.c +1 -0
index ba07e4a4ae3ffa..84cbba0ab698aa 100644
@@ -2291,6 +2291,7 @@ xfs_init_fs_context(
 	mp->m_allocsize_log = 16; /* 64k */
 
 	xfs_hooks_init(&mp->m_dir_update_hooks);
+	xfs_hooks_init(&mp->m_health_update_hooks);
 
 	fc->s_fs_info = mp;
 	fc->ops = &xfs_context_ops;
#8 lore raw

PATCH 5/22 xfs: create a filesystem shutdown hook

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create a hook so that health monitoring can report filesystem shutdown events to userspace. Shutdowns should be infrequent, so we don't bother with a static key here.

---
 fs/xfs/xfs_fsops.h |   11 +++++++++++
 fs/xfs/xfs_mount.h |    3 +++
 fs/xfs/xfs_fsops.c |   42 ++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_super.c |    1 +
 4 files changed, 57 insertions(+)
fs/xfs/xfs_fsops.h +11 -0
index 9d23c361ef56e4..ea5561b8580574 100644
@@ -15,4 +15,15 @@ int xfs_fs_goingdown(struct xfs_mount *mp, uint32_t inflags);
 int xfs_fs_reserve_ag_blocks(struct xfs_mount *mp);
 void xfs_fs_unreserve_ag_blocks(struct xfs_mount *mp);
 
+#ifdef CONFIG_XFS_LIVE_HOOKS
+struct xfs_shutdown_hook {
+	struct xfs_hook			shutdown_hook;
+};
+
+int xfs_shutdown_hook_add(struct xfs_mount *mp, struct xfs_shutdown_hook *hook);
+void xfs_shutdown_hook_del(struct xfs_mount *mp, struct xfs_shutdown_hook *hook);
+void xfs_shutdown_hook_setup(struct xfs_shutdown_hook *hook,
+		notifier_fn_t mod_fn);
+#endif /* CONFIG_XFS_LIVE_HOOKS */
+
 #endif	/* __XFS_FSOPS_H__ */
fs/xfs/xfs_mount.h +3 -0
index 3f20baaf9cc226..2d4305d91a3cd9 100644
@@ -346,6 +346,9 @@ typedef struct xfs_mount {
 
 	/* Hook to feed health events to a daemon. */
 	struct xfs_hooks	m_health_update_hooks;
+
+	/* Hook to feed shutdown events to a daemon. */
+	struct xfs_hooks	m_shutdown_hooks;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_fsops.c +42 -0

54 lines not on this page · raw message

fs/xfs/xfs_super.c +1 -0
index 84cbba0ab698aa..599900b9b0dd63 100644
@@ -2291,6 +2291,7 @@ xfs_init_fs_context(
 	mp->m_allocsize_log = 16; /* 64k */
 
 	xfs_hooks_init(&mp->m_dir_update_hooks);
+	xfs_hooks_init(&mp->m_shutdown_hooks);
 	xfs_hooks_init(&mp->m_health_update_hooks);
 
 	fc->s_fs_info = mp;
#9 lore raw

PATCH 6/22 xfs: create hooks for media errors

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Set up a media error event hook so that we can send events to userspace. Media errors are not expected to be frequent, so we don't have a static key guarding them here either.

---
 fs/xfs/xfs_mount.h          |    3 ++
 fs/xfs/xfs_notify_failure.h |   33 +++++++++++++++++++++
 fs/xfs/xfs_notify_failure.c |   68 ++++++++++++++++++++++++++++++++++++++++---
 fs/xfs/xfs_super.c          |    1 +
 4 files changed, 100 insertions(+), 5 deletions(-)
fs/xfs/xfs_mount.h +3 -0
index 2d4305d91a3cd9..0feb0fb685f51f 100644
@@ -349,6 +349,9 @@ typedef struct xfs_mount {
 
 	/* Hook to feed shutdown events to a daemon. */
 	struct xfs_hooks	m_shutdown_hooks;
+
+	/* Hook to feed media error events to a daemon. */
+	struct xfs_hooks	m_media_error_hooks;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_notify_failure.h +33 -0
index 8d08ec29dd2949..2695732ec20875 100644
@@ -8,4 +8,37 @@
 
 extern const struct dax_holder_operations xfs_dax_holder_operations;
 
+enum xfs_failed_device {
+	XFS_FAILED_DATADEV,
+	XFS_FAILED_LOGDEV,
+	XFS_FAILED_RTDEV,
+};
+
+#if defined(CONFIG_XFS_LIVE_HOOKS) && defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX)
+struct xfs_media_error_params {
+	struct xfs_mount		*mp;
+	enum xfs_failed_device		fdev;
+	xfs_daddr_t			daddr;
+	uint64_t			bbcount;
+	bool				pre_remove;
+};
+
+struct xfs_media_error_hook {
+	struct xfs_hook			error_hook;
+};
+
+int xfs_media_error_hook_add(struct xfs_mount *mp,
+		struct xfs_media_error_hook *hook);
+void xfs_media_error_hook_del(struct xfs_mount *mp,
+		struct xfs_media_error_hook *hook);
+void xfs_media_error_hook_setup(struct xfs_media_error_hook *hook,
+		notifier_fn_t mod_fn);
+#else
+struct xfs_media_error_params { };
+struct xfs_media_error_hook { };
+# define xfs_media_error_hook_add(...)		(0)
+# define xfs_media_error_hook_del(...)		((void)0)
+# define xfs_media_error_hook_setup(...)	((void)0)
+#endif /* CONFIG_XFS_LIVE_HOOKS */
+
 #endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_notify_failure.c +63 -5

91 lines not on this page · raw message

fs/xfs/xfs_super.c +1 -0
index 599900b9b0dd63..fb72a4976e8570 100644
@@ -2293,6 +2293,7 @@ xfs_init_fs_context(
 	xfs_hooks_init(&mp->m_dir_update_hooks);
 	xfs_hooks_init(&mp->m_shutdown_hooks);
 	xfs_hooks_init(&mp->m_health_update_hooks);
+	xfs_hooks_init(&mp->m_media_error_hooks);
 
 	fc->s_fs_info = mp;
 	fc->ops = &xfs_context_ops;
#10 lore raw

PATCH 7/22 iomap: report buffered read and write io errors to the filesystem

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Provide a callback so that iomap can report read and write IO errors to the caller filesystem. For now this is only wired up for iomap as a testbed for XFS.

---
 fs/iomap/internal.h               |    2 ++
 include/linux/fs.h                |    4 ++++
 Documentation/filesystems/vfs.rst |    7 +++++++
 fs/iomap/buffered-io.c            |   27 +++++++++++++++++++++++++--
 fs/iomap/ioend.c                  |    4 ++++
 5 files changed, 42 insertions(+), 2 deletions(-)
fs/iomap/internal.h +2 -0
index d05cb3aed96e79..06d9145b6be4fa 100644
@@ -5,5 +5,7 @@
 #define IOEND_BATCH_SIZE	4096
 
 u32 iomap_finish_ioend_direct(struct iomap_ioend *ioend);
+void iomap_mapping_ioerror(struct address_space *mapping, int direction,
+		loff_t pos, u64 len, int error);
 
 #endif /* _IOMAP_INTERNAL_H */
include/linux/fs.h +4 -0
index c895146c1444be..5e4b3a4b24823f 100644
@@ -477,6 +477,10 @@ struct address_space_operations {
 				sector_t *span);
 	void (*swap_deactivate)(struct file *file);
 	int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
+
+	/* Callback for dealing with IO errors during readahead or writeback */
+	void (*ioerror)(struct address_space *mapping, int direction,
+			loff_t pos, u64 len, int error);
 };
 
 extern const struct address_space_operations empty_aops;
Documentation/filesystems/vfs.rst +7 -0
index 4f13b01e42eb5e..9e70006bf99a63 100644
@@ -822,6 +822,8 @@ cache in your filesystem.  The following members are defined:
 		int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span)
 		int (*swap_deactivate)(struct file *);
 		int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
+		void (*ioerror)(struct address_space *mapping, int direction,
+				loff_t pos, u64 len, int error);
 	};
 
 ``read_folio``
@@ -1032,6 +1034,11 @@ cache in your filesystem.  The following members are defined:
 ``swap_rw``
 	Called to read or write swap pages when SWP_FS_OPS is set.
 
+``ioerror``
+        Called to deal with IO errors during readahead or writeback.
+        This may be called from interrupt context, and without any
+        locks necessarily being held.
+
 The File Object
 ===============
 
fs/iomap/buffered-io.c +25 -2
index 8b847a1e27f13e..8dd5421cb910b5 100644
@@ -288,6 +288,14 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
 		pos >= i_size_read(iter->inode);
 }
 
+inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,
+		loff_t pos, u64 len, int error)
+{
+	if (mapping && mapping->a_ops->ioerror)
+		mapping->a_ops->ioerror(mapping, direction, pos, len,
+				error);
+}
+
 /**
  * iomap_read_inline_data - copy inline data into the page cache
  * @iter: iteration structure
@@ -310,8 +318,11 @@ static int iomap_read_inline_data(const struct iomap_iter *iter,
 	if (folio_test_uptodate(folio))
 		return 0;
 
-	if (WARN_ON_ONCE(size > iomap->length))
+	if (WARN_ON_ONCE(size > iomap->length)) {
+		iomap_mapping_ioerror(folio->mapping, READ, iomap->offset,
+				size, -EIO);
 		return -EIO;
+	}
 	if (offset > 0)
 		ifs_alloc(iter->inode, folio, iter->flags);
 
@@ -339,6 +350,10 @@ static void iomap_finish_folio_read(struct folio *folio, size_t off,
 		spin_unlock_irqrestore(&ifs->state_lock, flags);
 	}
 
+	if (error)
+		iomap_mapping_ioerror(folio->mapping, READ,
+				folio_pos(folio) + off, len, error);
+
 	if (finished)
 		folio_end_read(folio, uptodate);
 }
@@ -558,11 +573,15 @@ static int iomap_read_folio_range(const struct iomap_iter *iter,
 	const struct iomap *srcmap = iomap_iter_srcmap(iter);
 	struct bio_vec bvec;
 	struct bio bio;
+	int ret;
 
 	bio_init(&bio, srcmap->bdev, &bvec, 1, REQ_OP_READ);
 	bio.bi_iter.bi_sector = iomap_sector(srcmap, pos);
 	bio_add_folio_nofail(&bio, folio, len, offset_in_folio(folio, pos));
-	return submit_bio_wait(&bio);
+	ret = submit_bio_wait(&bio);
+	if (ret)
+		iomap_mapping_ioerror(folio->mapping, READ, pos, len, ret);
+	return ret;
 }
 #else
 static int iomap_read_folio_range(const struct iomap_iter *iter,
@@ -1674,6 +1693,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)
 	u64 pos = folio_pos(folio);
 	u64 end_pos = pos + folio_size(folio);
 	u64 end_aligned = 0;
+	loff_t orig_pos = pos;
 	bool wb_pending = false;
 	int error = 0;
 	u32 rlen;
@@ -1724,6 +1744,9 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)
 
 	if (wb_pending)
 		wpc->nr_folios++;
+	if (error && pos > orig_pos)
+		iomap_mapping_ioerror(inode->i_mapping, WRITE, orig_pos, 0,
+				error);
 
 	/*
 	 * We can have dirty bits set past end of file in page_mkwrite path
fs/iomap/ioend.c +4 -0
index b49fa75eab260a..56e654f2d36fe9 100644
@@ -55,6 +55,10 @@ static u32 iomap_finish_ioend_buffered(struct iomap_ioend *ioend)
 
 	/* walk all folios in bio, ending page IO on them */
 	bio_for_each_folio_all(fi, bio) {
+		if (ioend->io_error)
+			iomap_mapping_ioerror(inode->i_mapping, WRITE,
+					folio_pos(fi.folio) + fi.offset,
+					fi.length, ioend->io_error);
 		iomap_finish_folio_write(inode, fi.folio, fi.length);
 		folio_count++;
 	}
#11 lore raw

PATCH 8/22 iomap: report directio read and write errors to callers

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Add more hooks to report directio IO errors to the filesystem.

---
 include/linux/iomap.h |    2 ++
 fs/iomap/direct-io.c  |    4 ++++
 2 files changed, 6 insertions(+)
include/linux/iomap.h +2 -0
index 73dceabc21c8c7..ca1590e5002342 100644
@@ -486,6 +486,8 @@ struct iomap_dio_ops {
 		      unsigned flags);
 	void (*submit_io)(const struct iomap_iter *iter, struct bio *bio,
 		          loff_t file_offset);
+	void (*ioerror)(struct inode *inode, int direction, loff_t pos,
+			u64 len, int error);
 
 	/*
 	 * Filesystems wishing to attach private information to a direct io bio
fs/iomap/direct-io.c +4 -0
index 5d5d63efbd5767..1512d8dbb0d2e7 100644
@@ -95,6 +95,10 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 
 	if (dops && dops->end_io)
 		ret = dops->end_io(iocb, dio->size, ret, dio->flags);
+	if (dio->error && dops && dops->ioerror)
+		dops->ioerror(file_inode(iocb->ki_filp),
+				(dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
+				offset, dio->size, dio->error);
 
 	if (likely(!ret)) {
 		ret = dio->size;
#12 lore raw

PATCH 9/22 xfs: create file io error hooks

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create hooks within XFS to deliver IO errors to callers. File I/O errors are usually rare, so we don't employ a static key here.

---
 fs/xfs/xfs_file.h           |   37 +++++++++
 fs/xfs/xfs_mount.h          |    3 +
 fs/xfs/xfs_aops.c           |    2 
 fs/xfs/xfs_file.c           |  174 +++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_notify_failure.c |    5 +
 fs/xfs/xfs_super.c          |    1 
 6 files changed, 221 insertions(+), 1 deletion(-)
fs/xfs/xfs_file.h +37 -0
index 2ad91f755caf35..441f8a693bb884 100644
@@ -12,4 +12,41 @@ extern const struct file_operations xfs_dir_file_operations;
 bool xfs_is_falloc_aligned(struct xfs_inode *ip, loff_t pos,
 		long long int len);
 
+enum xfs_file_ioerror_type {
+	XFS_FILE_IOERROR_BUFFERED_READ,
+	XFS_FILE_IOERROR_BUFFERED_WRITE,
+	XFS_FILE_IOERROR_DIRECT_READ,
+	XFS_FILE_IOERROR_DIRECT_WRITE,
+	XFS_FILE_IOERROR_DATA_LOST,
+};
+
+struct xfs_file_ioerror_params {
+	xfs_ino_t		ino;
+	loff_t			pos;
+	u64			len;
+	u32			gen;
+	int			error;
+};
+
+#ifdef CONFIG_XFS_LIVE_HOOKS
+struct xfs_file_ioerror_hook {
+	struct xfs_hook			ioerror_hook;
+};
+
+int xfs_file_ioerror_hook_add(struct xfs_mount *mp,
+		struct xfs_file_ioerror_hook *hook);
+void xfs_file_ioerror_hook_del(struct xfs_mount *mp,
+		struct xfs_file_ioerror_hook *hook);
+void xfs_file_ioerror_hook_setup(struct xfs_file_ioerror_hook *hook,
+		notifier_fn_t mod_fn);
+
+void xfs_vm_ioerror(struct address_space *mapping, int direction, loff_t pos,
+		u64 len, int error);
+
+void xfs_inode_media_error(struct xfs_inode *ip, loff_t pos, u64 len);
+#else
+# define xfs_vm_ioerror			NULL
+# define xfs_inode_media_error(...)	((void)0)
+#endif /* CONFIG_XFS_LIVE_HOOKS */
+
 #endif /* __XFS_FILE_H__ */
fs/xfs/xfs_mount.h +3 -0
index 0feb0fb685f51f..2d7f9ccba5287e 100644
@@ -352,6 +352,9 @@ typedef struct xfs_mount {
 
 	/* Hook to feed media error events to a daemon. */
 	struct xfs_hooks	m_media_error_hooks;
+
+	/* Hook to feed file io error events to a daemon. */
+	struct xfs_hooks	m_file_ioerror_hooks;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_aops.c +2 -0
index a26f798155331f..f3f28b9ae0f70e 100644
@@ -22,6 +22,7 @@
 #include "xfs_icache.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_rtgroup.h"
+#include "xfs_file.h"
 
 struct xfs_writepage_ctx {
 	struct iomap_writepage_ctx ctx;
@@ -810,6 +811,7 @@ const struct address_space_operations xfs_address_space_operations = {
 	.is_partially_uptodate  = iomap_is_partially_uptodate,
 	.error_remove_folio	= generic_error_remove_folio,
 	.swap_activate		= xfs_vm_swap_activate,
+	.ioerror		= xfs_vm_ioerror,
 };
 
 const struct address_space_operations xfs_dax_aops = {
fs/xfs/xfs_file.c +173 -1

192 lines not on this page · raw message

fs/xfs/xfs_notify_failure.c +5 -0
index 557f4bf3463dcb..8766d83385ddad 100644
@@ -22,6 +22,7 @@
 #include "xfs_notify_failure.h"
 #include "xfs_rtgroup.h"
 #include "xfs_rtrmap_btree.h"
+#include "xfs_file.h"
 
 #include <linux/mm.h>
 #include <linux/dax.h>
@@ -167,6 +168,10 @@ xfs_dax_failure_fn(
 		invalidate_inode_pages2_range(mapping, pgoff,
 					      pgoff + pgcnt - 1);
 
+	xfs_inode_media_error(ip,
+			XFS_FSB_TO_B(mp, (u64)pgoff << PAGE_SHIFT),
+			XFS_FSB_TO_B(mp, (u64)pgcnt << PAGE_SHIFT));
+
 	xfs_irele(ip);
 	return error;
 }
fs/xfs/xfs_super.c +1 -0
index fb72a4976e8570..54d82f5a5b8863 100644
@@ -2294,6 +2294,7 @@ xfs_init_fs_context(
 	xfs_hooks_init(&mp->m_shutdown_hooks);
 	xfs_hooks_init(&mp->m_health_update_hooks);
 	xfs_hooks_init(&mp->m_media_error_hooks);
+	xfs_hooks_init(&mp->m_file_ioerror_hooks);
 
 	fc->s_fs_info = mp;
 	fc->ops = &xfs_context_ops;
#13 lore raw

PATCH 10/22 xfs: create a special file to pass filesystem health to userspace

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create an ioctl that installs a file descriptor backed by an anon_inode file that will convey filesystem health events to userspace.

---
 fs/xfs/libxfs/xfs_fs.h |    8 ++
 fs/xfs/xfs_healthmon.h |   16 +++++
 fs/xfs/Kconfig         |    8 ++
 fs/xfs/Makefile        |    1 
 fs/xfs/xfs_healthmon.c |  157 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_ioctl.c     |    4 +
 6 files changed, 194 insertions(+)
 create mode 100644 fs/xfs/xfs_healthmon.h
 create mode 100644 fs/xfs/xfs_healthmon.c
fs/xfs/libxfs/xfs_fs.h +8 -0
index 12463ba766da05..dba7896f716092 100644
@@ -1003,6 +1003,13 @@ struct xfs_rtgroup_geometry {
 #define XFS_RTGROUP_GEOM_SICK_RMAPBT	(1U << 3)  /* reverse mappings */
 #define XFS_RTGROUP_GEOM_SICK_REFCNTBT	(1U << 4)  /* reference counts */
 
+struct xfs_health_monitor {
+	__u64	flags;		/* flags */
+	__u8	format;		/* output format */
+	__u8	pad1[7];	/* zeroes */
+	__u64	pad2[2];	/* zeroes */
+};
+
 /*
  * ioctl commands that are used by Linux filesystems
  */
@@ -1042,6 +1049,7 @@ struct xfs_rtgroup_geometry {
 #define XFS_IOC_GETPARENTS_BY_HANDLE _IOWR('X', 63, struct xfs_getparents_by_handle)
 #define XFS_IOC_SCRUBV_METADATA	_IOWR('X', 64, struct xfs_scrub_vec_head)
 #define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)
+#define XFS_IOC_HEALTH_MONITOR	_IOW ('X', 68, struct xfs_health_monitor)
 
 /*
  * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.h new file +16 -0
new file mode 100644
index 00000000000000..07126e39281a0c
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+ * Author: Darrick J. Wong <[email protected]>
+ */
+#ifndef __XFS_HEALTHMON_H__
+#define __XFS_HEALTHMON_H__
+
+#ifdef CONFIG_XFS_HEALTH_MONITOR
+long xfs_ioc_health_monitor(struct xfs_mount *mp,
+		struct xfs_health_monitor __user *arg);
+#else
+# define xfs_ioc_health_monitor(mp, hmo)	(-ENOTTY)
+#endif /* CONFIG_XFS_HEALTH_MONITOR */
+
+#endif /* __XFS_HEALTHMON_H__ */
fs/xfs/Kconfig +8 -0
index b99da294e9a310..682d9a35203494 100644
@@ -130,6 +130,14 @@ config XFS_RT
 
 	  If unsure, say N.
 
+config XFS_HEALTH_MONITOR
+	bool "Report filesystem health events to userspace"
+	depends on XFS_FS
+	select XFS_LIVE_HOOKS
+	default y
+	help
+	  Report health events to userspace programs.
+
 config XFS_DRAIN_INTENTS
 	bool
 	select JUMP_LABEL if HAVE_ARCH_JUMP_LABEL
fs/xfs/Makefile +1 -0
index 5bf501cf827172..d4e9070a9326ba 100644
@@ -157,6 +157,7 @@ xfs-$(CONFIG_XFS_DRAIN_INTENTS)	+= xfs_drain.o
 xfs-$(CONFIG_XFS_LIVE_HOOKS)	+= xfs_hooks.o
 xfs-$(CONFIG_XFS_MEMORY_BUFS)	+= xfs_buf_mem.o
 xfs-$(CONFIG_XFS_BTREE_IN_MEM)	+= libxfs/xfs_btree_mem.o
+xfs-$(CONFIG_XFS_HEALTH_MONITOR) += xfs_healthmon.o
 
 # online scrub/repair
 ifeq ($(CONFIG_XFS_ONLINE_SCRUB),y)
fs/xfs/xfs_healthmon.c new file +157 -0

157 lines not on this page · raw message

fs/xfs/xfs_ioctl.c +4 -0
index a6bb7ee7a27ad5..08998d84554f09 100644
@@ -41,6 +41,7 @@
 #include "xfs_exchrange.h"
 #include "xfs_handle.h"
 #include "xfs_rtgroup.h"
+#include "xfs_healthmon.h"
 
 #include <linux/mount.h>
 #include <linux/fileattr.h>
@@ -1421,6 +1422,9 @@ xfs_file_ioctl(
 	case XFS_IOC_COMMIT_RANGE:
 		return xfs_ioc_commit_range(filp, arg);
 
+	case XFS_IOC_HEALTH_MONITOR:
+		return xfs_ioc_health_monitor(mp, arg);
+
 	default:
 		return -ENOTTY;
 	}
#14 lore raw

PATCH 11/22 xfs: create event queuing, formatting, and discovery infrastructure

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create the basic infrastructure that we need to report health events to userspace. We need a compact form for recording critical information about an event and queueing them; a means to notice that we've lost some events; and a means to format the events into something that userspace can handle. Make the kernel export C structures via read().

In a previous iteration of this new subsystem, I wanted to explore data exchange formats that are more flexible and easier for humans to read than C structures. The thought being that when we want to rev (or worse, enlarge) the event format, it ought to be trivially easy to do that in a way that doesn't break old userspace.

I looked at formats such as protobufs and capnproto. These look really nice in that extending the wire format is fairly easy, you can give it a data schema and it generates the serialization code for you, handles endianness problems, etc. The huge downside is that neither support C all that well.

Too hard, and didn't want to port either of those huge sprawling libraries first to the kernel and then again to xfsprogs. Then I thought, how about JSON? Javascript objects are human readable, the kernel can emit json without much fuss (it's all just strings!) and there are plenty of interpreters for python/rust/c/etc.

There's a proposed schema format for json, which means that xfs can publish a description of the events that kernel will emit. Userspace consumers (e.g. xfsprogs/xfs_healer) can embed the same schema document and use it to validate the incoming events from the kernel, which means it can discard events that it doesn't understand, or garbage being emitted due to bugs.

However, json has a huge crutch -- javascript is well known for its vague definitions of what are numbers. This makes expressing a large number rather fraught, because the runtime is free to represent a number in nearly any way it wants. Stupider ones will truncate values to word size, others will roll out doubles for uint52_t (yes, fifty-two) with the resulting loss of precision. Not good when you're dealing with discrete units.

It just so happens that python's json library is smart enough to see a sequence of digits and put them in a u64 (at least on x86_64/aarch64) but an actual javascript interpreter (pasting into Firefox) isn't necessarily so clever.

It turns out that none of the proposed json schemas were ever ratified even in an open-consensus way, so json blobs are still just loosely structured blobs. The parsing in userspace was also noticeably slow and memory-consumptive.

Hence only the C interface survives.

---
 fs/xfs/libxfs/xfs_fs.h |   47 ++++
 fs/xfs/xfs_healthmon.h |   29 +++
 fs/xfs/xfs_linux.h     |    3 
 fs/xfs/xfs_trace.h     |  170 +++++++++++++++
 fs/xfs/xfs_healthmon.c |  542 +++++++++++++++++++++++++++++++++++++++++++++++-
 fs/xfs/xfs_trace.c     |    2 
 lib/seq_buf.c          |    1 
 7 files changed, 787 insertions(+), 7 deletions(-)
fs/xfs/libxfs/xfs_fs.h +47 -0
index dba7896f716092..dfca42b2c31192 100644
@@ -1003,6 +1003,45 @@ struct xfs_rtgroup_geometry {
 #define XFS_RTGROUP_GEOM_SICK_RMAPBT	(1U << 3)  /* reverse mappings */
 #define XFS_RTGROUP_GEOM_SICK_REFCNTBT	(1U << 4)  /* reference counts */
 
+/* Health monitor event domains */
+
+/* affects the whole fs */
+#define XFS_HEALTH_MONITOR_DOMAIN_MOUNT		(0)
+
+/* Health monitor event types */
+
+/* status of the monitor itself */
+#define XFS_HEALTH_MONITOR_TYPE_RUNNING		(0)
+#define XFS_HEALTH_MONITOR_TYPE_LOST		(1)
+
+/* lost events */
+struct xfs_health_monitor_lost {
+	__u64	count;
+};
+
+struct xfs_health_monitor_event {
+	/* XFS_HEALTH_MONITOR_DOMAIN_* */
+	__u32	domain;
+
+	/* XFS_HEALTH_MONITOR_TYPE_* */
+	__u32	type;
+
+	/* Timestamp of the event, in nanoseconds since the Unix epoch */
+	__u64	time_ns;
+
+	/*
+	 * Details of the event.  The primary clients are written in python
+	 * and rust, so break this up because bindgen hates anonymous structs
+	 * and unions.
+	 */
+	union {
+		struct xfs_health_monitor_lost lost;
+	} e;
+
+	/* zeroes */
+	__u64	pad[2];
+};
+
 struct xfs_health_monitor {
 	__u64	flags;		/* flags */
 	__u8	format;		/* output format */
@@ -1010,6 +1049,14 @@ struct xfs_health_monitor {
 	__u64	pad2[2];	/* zeroes */
 };
 
+/* Return all health status events, not just deltas */
+#define XFS_HEALTH_MONITOR_VERBOSE	(1ULL << 0)
+
+#define XFS_HEALTH_MONITOR_ALL		(XFS_HEALTH_MONITOR_VERBOSE)
+
+/* Initial return format version */
+#define XFS_HEALTH_MONITOR_FMT_V0	(0)
+
 /*
  * ioctl commands that are used by Linux filesystems
  */
fs/xfs/xfs_healthmon.h +29 -0
index 07126e39281a0c..ea2d6a327dfb16 100644
@@ -6,6 +6,35 @@
 #ifndef __XFS_HEALTHMON_H__
 #define __XFS_HEALTHMON_H__
 
+enum xfs_healthmon_type {
+	XFS_HEALTHMON_RUNNING,	/* monitor running */
+	XFS_HEALTHMON_LOST,	/* message lost */
+};
+
+enum xfs_healthmon_domain {
+	XFS_HEALTHMON_MOUNT,	/* affects the whole fs */
+};
+
+struct xfs_healthmon_event {
+	struct xfs_healthmon_event	*next;
+
+	enum xfs_healthmon_type		type;
+	enum xfs_healthmon_domain	domain;
+
+	uint64_t			time_ns;
+
+	union {
+		/* lost events */
+		struct {
+			uint64_t	lostcount;
+		};
+		/* mount */
+		struct {
+			unsigned int	flags;
+		};
+	};
+};
+
 #ifdef CONFIG_XFS_HEALTH_MONITOR
 long xfs_ioc_health_monitor(struct xfs_mount *mp,
 		struct xfs_health_monitor __user *arg);
fs/xfs/xfs_linux.h +3 -0
index 4dd747bdbccab2..e122db938cc06b 100644
@@ -63,6 +63,9 @@ typedef __u32			xfs_nlink_t;
 #include <linux/xattr.h>
 #include <linux/mnt_idmapping.h>
 #include <linux/debugfs.h>
+#ifdef CONFIG_XFS_HEALTH_MONITOR
+# include <linux/seq_buf.h>
+#endif
 
 #include <asm/page.h>
 #include <asm/div64.h>
fs/xfs/xfs_trace.h +170 -0

182 lines not on this page · raw message

fs/xfs/xfs_healthmon.c +535 -7

611 lines not on this page · raw message

fs/xfs/xfs_trace.c +2 -0
index a60556dbd172ee..d42b864a3837a2 100644
@@ -51,6 +51,8 @@
 #include "xfs_rtgroup.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_zone_priv.h"
+#include "xfs_health.h"
+#include "xfs_healthmon.h"
 
 /*
  * We include this last to have the helpers above available for the trace
lib/seq_buf.c +1 -0
index f3f3436d60a940..f6a1fb46a1d6c9 100644
@@ -245,6 +245,7 @@ int seq_buf_putmem(struct seq_buf *s, const void *mem, unsigned int len)
 	seq_buf_set_overflow(s);
 	return -1;
 }
+EXPORT_SYMBOL_GPL(seq_buf_putmem);
 
 #define MAX_MEMHEX_BYTES	8U
 #define HEX_CHARS		(MAX_MEMHEX_BYTES*2 + 1)
#15 lore raw

PATCH 12/22 xfs: report metadata health events through healthmon

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Set up a metadata health event hook so that we can send events to userspace as we collect information. The unmount hook severs the weak reference between the health monitor and the filesystem it's monitoring; when this happens, we stop reporting events because there's no longer any point.

---
 fs/xfs/libxfs/xfs_fs.h     |   38 +++++
 fs/xfs/libxfs/xfs_health.h |    5 +
 fs/xfs/xfs_healthmon.h     |   31 ++++
 fs/xfs/xfs_trace.h         |   98 +++++++++++++
 fs/xfs/xfs_health.c        |   67 +++++++++
 fs/xfs/xfs_healthmon.c     |  333 +++++++++++++++++++++++++++++++++++++++++++-
 6 files changed, 563 insertions(+), 9 deletions(-)
fs/xfs/libxfs/xfs_fs.h +38 -0
index dfca42b2c31192..2ad45351ac0ea6 100644
@@ -1008,17 +1008,52 @@ struct xfs_rtgroup_geometry {
 /* affects the whole fs */
 #define XFS_HEALTH_MONITOR_DOMAIN_MOUNT		(0)
 
+/* metadata health events */
+#define XFS_HEALTH_MONITOR_DOMAIN_FS		(1)
+#define XFS_HEALTH_MONITOR_DOMAIN_AG		(2)
+#define XFS_HEALTH_MONITOR_DOMAIN_INODE		(3)
+#define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP	(4)
+
 /* Health monitor event types */
 
 /* status of the monitor itself */
 #define XFS_HEALTH_MONITOR_TYPE_RUNNING		(0)
 #define XFS_HEALTH_MONITOR_TYPE_LOST		(1)
 
+/* metadata health events */
+#define XFS_HEALTH_MONITOR_TYPE_SICK		(2)
+#define XFS_HEALTH_MONITOR_TYPE_CORRUPT		(3)
+#define XFS_HEALTH_MONITOR_TYPE_HEALTHY		(4)
+
+/* filesystem was unmounted */
+#define XFS_HEALTH_MONITOR_TYPE_UNMOUNT		(5)
+
 /* lost events */
 struct xfs_health_monitor_lost {
 	__u64	count;
 };
 
+/* fs/rt metadata */
+struct xfs_health_monitor_fs {
+	/* XFS_FSOP_GEOM_SICK_* flags */
+	__u32	mask;
+};
+
+/* ag/rtgroup metadata */
+struct xfs_health_monitor_group {
+	/* XFS_{AG,RTGROUP}_SICK_* flags */
+	__u32	mask;
+	__u32	gno;
+};
+
+/* inode metadata */
+struct xfs_health_monitor_inode {
+	/* XFS_BS_SICK_* flags */
+	__u32	mask;
+	__u32	gen;
+	__u64	ino;
+};
+
 struct xfs_health_monitor_event {
 	/* XFS_HEALTH_MONITOR_DOMAIN_* */
 	__u32	domain;
@@ -1036,6 +1071,9 @@ struct xfs_health_monitor_event {
 	 */
 	union {
 		struct xfs_health_monitor_lost lost;
+		struct xfs_health_monitor_fs fs;
+		struct xfs_health_monitor_group group;
+		struct xfs_health_monitor_inode inode;
 	} e;
 
 	/* zeroes */
fs/xfs/libxfs/xfs_health.h +5 -0
index 39fef33dedc6a8..9ff3bf8ba4ed8f 100644
@@ -336,4 +336,9 @@ void xfs_health_hook_del(struct xfs_mount *mp, struct xfs_health_hook *hook);
 void xfs_health_hook_setup(struct xfs_health_hook *hook, notifier_fn_t mod_fn);
 #endif /* CONFIG_XFS_LIVE_HOOKS */
 
+unsigned int xfs_healthmon_inode_mask(unsigned int sick_mask);
+unsigned int xfs_healthmon_rtgroup_mask(unsigned int sick_mask);
+unsigned int xfs_healthmon_perag_mask(unsigned int sick_mask);
+unsigned int xfs_healthmon_fs_mask(unsigned int sick_mask);
+
 #endif	/* __XFS_HEALTH_H__ */
fs/xfs/xfs_healthmon.h +31 -0
index ea2d6a327dfb16..3f3ba16d5af56a 100644
@@ -9,10 +9,23 @@
 enum xfs_healthmon_type {
 	XFS_HEALTHMON_RUNNING,	/* monitor running */
 	XFS_HEALTHMON_LOST,	/* message lost */
+	XFS_HEALTHMON_UNMOUNT,	/* filesystem is unmounting */
+
+	/* metadata health events */
+	XFS_HEALTHMON_SICK,	/* runtime corruption observed */
+	XFS_HEALTHMON_CORRUPT,	/* fsck reported corruption */
+	XFS_HEALTHMON_HEALTHY,	/* fsck reported healthy structure */
+
 };
 
 enum xfs_healthmon_domain {
 	XFS_HEALTHMON_MOUNT,	/* affects the whole fs */
+
+	/* metadata health events */
+	XFS_HEALTHMON_FS,	/* main filesystem metadata */
+	XFS_HEALTHMON_AG,	/* allocation group metadata */
+	XFS_HEALTHMON_INODE,	/* inode metadata */
+	XFS_HEALTHMON_RTGROUP,	/* realtime group metadata */
 };
 
 struct xfs_healthmon_event {
@@ -32,6 +45,24 @@ struct xfs_healthmon_event {
 		struct {
 			unsigned int	flags;
 		};
+		/* fs/rt metadata */
+		struct {
+			/* XFS_SICK_* flags */
+			unsigned int	fsmask;
+		};
+		/* ag/rtgroup metadata */
+		struct {
+			/* XFS_SICK_* flags */
+			unsigned int	grpmask;
+			unsigned int	group;
+		};
+		/* inode metadata */
+		struct {
+			/* XFS_SICK_INO_* flags */
+			unsigned int	imask;
+			uint32_t	gen;
+			xfs_ino_t	ino;
+		};
 	};
 };
 
fs/xfs/xfs_trace.h +96 -2

127 lines not on this page · raw message

fs/xfs/xfs_health.c +67 -0

91 lines not on this page · raw message

fs/xfs/xfs_healthmon.c +326 -7

447 lines not on this page · raw message

#16 lore raw

PATCH 13/22 xfs: report shutdown events through healthmon

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Set up a shutdown hook so that we can send notifications to userspace.

---
 fs/xfs/libxfs/xfs_fs.h |   18 +++++++++
 fs/xfs/xfs_healthmon.h |    5 ++-
 fs/xfs/xfs_trace.h     |   28 ++++++++++++++
 fs/xfs/xfs_healthmon.c |   93 +++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 141 insertions(+), 3 deletions(-)
fs/xfs/libxfs/xfs_fs.h +18 -0
index 2ad45351ac0ea6..677141a17605a4 100644
@@ -1028,6 +1028,9 @@ struct xfs_rtgroup_geometry {
 /* filesystem was unmounted */
 #define XFS_HEALTH_MONITOR_TYPE_UNMOUNT		(5)
 
+/* filesystem shutdown */
+#define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN	(6)
+
 /* lost events */
 struct xfs_health_monitor_lost {
 	__u64	count;
@@ -1054,6 +1057,20 @@ struct xfs_health_monitor_inode {
 	__u64	ino;
 };
 
+/* shutdown reasons */
+#define XFS_HEALTH_SHUTDOWN_META_IO_ERROR	(1u << 0)
+#define XFS_HEALTH_SHUTDOWN_LOG_IO_ERROR	(1u << 1)
+#define XFS_HEALTH_SHUTDOWN_FORCE_UMOUNT	(1u << 2)
+#define XFS_HEALTH_SHUTDOWN_CORRUPT_INCORE	(1u << 3)
+#define XFS_HEALTH_SHUTDOWN_CORRUPT_ONDISK	(1u << 4)
+#define XFS_HEALTH_SHUTDOWN_DEVICE_REMOVED	(1u << 5)
+
+/* shutdown */
+struct xfs_health_monitor_shutdown {
+	/* XFS_HEALTH_SHUTDOWN_* flags */
+	__u32	reasons;
+};
+
 struct xfs_health_monitor_event {
 	/* XFS_HEALTH_MONITOR_DOMAIN_* */
 	__u32	domain;
@@ -1074,6 +1091,7 @@ struct xfs_health_monitor_event {
 		struct xfs_health_monitor_fs fs;
 		struct xfs_health_monitor_group group;
 		struct xfs_health_monitor_inode inode;
+		struct xfs_health_monitor_shutdown shutdown;
 	} e;
 
 	/* zeroes */
fs/xfs/xfs_healthmon.h +4 -1
index 3f3ba16d5af56a..a82a684bbc0e03 100644
@@ -11,6 +11,9 @@ enum xfs_healthmon_type {
 	XFS_HEALTHMON_LOST,	/* message lost */
 	XFS_HEALTHMON_UNMOUNT,	/* filesystem is unmounting */
 
+	/* filesystem shutdown */
+	XFS_HEALTHMON_SHUTDOWN,
+
 	/* metadata health events */
 	XFS_HEALTHMON_SICK,	/* runtime corruption observed */
 	XFS_HEALTHMON_CORRUPT,	/* fsck reported corruption */
@@ -41,7 +44,7 @@ struct xfs_healthmon_event {
 		struct {
 			uint64_t	lostcount;
 		};
-		/* mount */
+		/* shutdown */
 		struct {
 			unsigned int	flags;
 		};
fs/xfs/xfs_trace.h +28 -0
index 051599f8433ed6..b2b056ceb52f5c 100644
@@ -6009,8 +6009,32 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_read_finish);
 DEFINE_HEALTHMON_EVENT(xfs_healthmon_release);
 DEFINE_HEALTHMON_EVENT(xfs_healthmon_unmount);
 
+TRACE_EVENT(xfs_healthmon_shutdown_hook,
+	TP_PROTO(const struct xfs_mount *mp, uint32_t shutdown_flags,
+		 unsigned int events, unsigned long long lost_prev),
+	TP_ARGS(mp, shutdown_flags, events, lost_prev),
+	TP_STRUCT__entry(
+		__field(dev_t, dev)
+		__field(uint32_t, shutdown_flags)
+		__field(unsigned int, events)
+		__field(unsigned long long, lost_prev)
+	),
+	TP_fast_assign(
+		__entry->dev = mp ? mp->m_super->s_dev : 0;
+		__entry->shutdown_flags = shutdown_flags;
+		__entry->events = events;
+		__entry->lost_prev = lost_prev;
+	),
+	TP_printk("dev %d:%d shutdown_flags %s events %u lost_prev? %llu",
+		  MAJOR(__entry->dev), MINOR(__entry->dev),
+		  __print_flags(__entry->shutdown_flags, "|", XFS_SHUTDOWN_STRINGS),
+		  __entry->events,
+		  __entry->lost_prev)
+);
+
 #define XFS_HEALTHMON_TYPE_STRINGS \
 	{ XFS_HEALTHMON_LOST,		"lost" }, \
+	{ XFS_HEALTHMON_SHUTDOWN,	"shutdown" }, \
 	{ XFS_HEALTHMON_UNMOUNT,	"unmount" }, \
 	{ XFS_HEALTHMON_SICK,		"sick" }, \
 	{ XFS_HEALTHMON_CORRUPT,	"corrupt" }, \
@@ -6024,6 +6048,7 @@ DEFINE_HEALTHMON_EVENT(xfs_healthmon_unmount);
 	{ XFS_HEALTHMON_RTGROUP,	"rtgroup" }
 
 TRACE_DEFINE_ENUM(XFS_HEALTHMON_LOST);
+TRACE_DEFINE_ENUM(XFS_HEALTHMON_SHUTDOWN);
 TRACE_DEFINE_ENUM(XFS_HEALTHMON_UNMOUNT);
 TRACE_DEFINE_ENUM(XFS_HEALTHMON_SICK);
 TRACE_DEFINE_ENUM(XFS_HEALTHMON_CORRUPT);
@@ -6064,6 +6089,9 @@ DECLARE_EVENT_CLASS(xfs_healthmon_event_class,
 		switch (__entry->domain) {
 		case XFS_HEALTHMON_MOUNT:
 			switch (__entry->type) {
+			case XFS_HEALTHMON_SHUTDOWN:
+				__entry->mask = event->flags;
+				break;
 			case XFS_HEALTHMON_LOST:
 				__entry->lostcount = event->lostcount;
 				break;
fs/xfs/xfs_healthmon.c +91 -2

158 lines not on this page · raw message

#17 lore raw

PATCH 14/22 xfs: report media errors through healthmon

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Now that we have hooks to report media errors, connect this to the health monitor as well.

---
 fs/xfs/libxfs/xfs_fs.h |   15 +++++++++
 fs/xfs/xfs_healthmon.h |   12 +++++++
 fs/xfs/xfs_trace.h     |   57 ++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_healthmon.c |   77 +++++++++++++++++++++++++++++++++++++++++++++++-
 fs/xfs/xfs_trace.c     |    1 +
 5 files changed, 160 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +15 -0
index 677141a17605a4..0711007344e16d 100644
@@ -1014,6 +1014,11 @@ struct xfs_rtgroup_geometry {
 #define XFS_HEALTH_MONITOR_DOMAIN_INODE		(3)
 #define XFS_HEALTH_MONITOR_DOMAIN_RTGROUP	(4)
 
+/* disk events */
+#define XFS_HEALTH_MONITOR_DOMAIN_DATADEV	(5)
+#define XFS_HEALTH_MONITOR_DOMAIN_RTDEV		(6)
+#define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV	(7)
+
 /* Health monitor event types */
 
 /* status of the monitor itself */
@@ -1031,6 +1036,9 @@ struct xfs_rtgroup_geometry {
 /* filesystem shutdown */
 #define XFS_HEALTH_MONITOR_TYPE_SHUTDOWN	(6)
 
+/* media errors */
+#define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR	(7)
+
 /* lost events */
 struct xfs_health_monitor_lost {
 	__u64	count;
@@ -1071,6 +1079,12 @@ struct xfs_health_monitor_shutdown {
 	__u32	reasons;
 };
 
+/* disk media errors */
+struct xfs_health_monitor_media {
+	__u64	daddr;
+	__u64	bbcount;
+};
+
 struct xfs_health_monitor_event {
 	/* XFS_HEALTH_MONITOR_DOMAIN_* */
 	__u32	domain;
@@ -1092,6 +1106,7 @@ struct xfs_health_monitor_event {
 		struct xfs_health_monitor_group group;
 		struct xfs_health_monitor_inode inode;
 		struct xfs_health_monitor_shutdown shutdown;
+		struct xfs_health_monitor_media media;
 	} e;
 
 	/* zeroes */
fs/xfs/xfs_healthmon.h +12 -0
index a82a684bbc0e03..407c5e1f466726 100644
@@ -19,6 +19,8 @@ enum xfs_healthmon_type {
 	XFS_HEALTHMON_CORRUPT,	/* fsck reported corruption */
 	XFS_HEALTHMON_HEALTHY,	/* fsck reported healthy structure */
 
+	/* media errors */
+	XFS_HEALTHMON_MEDIA_ERROR,
 };
 
 enum xfs_healthmon_domain {
@@ -29,6 +31,11 @@ enum xfs_healthmon_domain {
 	XFS_HEALTHMON_AG,	/* allocation group metadata */
 	XFS_HEALTHMON_INODE,	/* inode metadata */
 	XFS_HEALTHMON_RTGROUP,	/* realtime group metadata */
+
+	/* media errors */
+	XFS_HEALTHMON_DATADEV,
+	XFS_HEALTHMON_RTDEV,
+	XFS_HEALTHMON_LOGDEV,
 };
 
 struct xfs_healthmon_event {
@@ -66,6 +73,11 @@ struct xfs_healthmon_event {
 			uint32_t	gen;
 			xfs_ino_t	ino;
 		};
+		/* media errors */
+		struct {
+			xfs_daddr_t	daddr;
+			uint64_t	bbcount;
+		};
 	};
 };
 
fs/xfs/xfs_trace.h +57 -0

75 lines not on this page · raw message

fs/xfs/xfs_healthmon.c +75 -2

142 lines not on this page · raw message

fs/xfs/xfs_trace.c +1 -0
index d42b864a3837a2..08ddab700a6cd3 100644
@@ -53,6 +53,7 @@
 #include "xfs_zone_priv.h"
 #include "xfs_health.h"
 #include "xfs_healthmon.h"
+#include "xfs_notify_failure.h"
 
 /*
  * We include this last to have the helpers above available for the trace
#18 lore raw

PATCH 15/22 xfs: report file io errors through healthmon

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Set up a file io error event hook so that we can send events about read errors, writeback errors, and directio errors to userspace.

---
 fs/xfs/libxfs/xfs_fs.h |   19 +++++++++
 fs/xfs/xfs_healthmon.h |   17 ++++++++
 fs/xfs/xfs_trace.h     |   59 +++++++++++++++++++++++++++++
 fs/xfs/xfs_healthmon.c |   98 +++++++++++++++++++++++++++++++++++++++++++++++-
 fs/xfs/xfs_trace.c     |    1 
 5 files changed, 192 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_fs.h +19 -0
index 0711007344e16d..a96f11d9bd9c64 100644
@@ -1019,6 +1019,9 @@ struct xfs_rtgroup_geometry {
 #define XFS_HEALTH_MONITOR_DOMAIN_RTDEV		(6)
 #define XFS_HEALTH_MONITOR_DOMAIN_LOGDEV	(7)
 
+/* file range events */
+#define XFS_HEALTH_MONITOR_DOMAIN_FILERANGE	(8)
+
 /* Health monitor event types */
 
 /* status of the monitor itself */
@@ -1039,6 +1042,13 @@ struct xfs_rtgroup_geometry {
 /* media errors */
 #define XFS_HEALTH_MONITOR_TYPE_MEDIA_ERROR	(7)
 
+/* file range events */
+#define XFS_HEALTH_MONITOR_TYPE_BUFREAD		(8)
+#define XFS_HEALTH_MONITOR_TYPE_BUFWRITE	(9)
+#define XFS_HEALTH_MONITOR_TYPE_DIOREAD		(10)
+#define XFS_HEALTH_MONITOR_TYPE_DIOWRITE	(11)
+#define XFS_HEALTH_MONITOR_TYPE_DATALOST	(12)
+
 /* lost events */
 struct xfs_health_monitor_lost {
 	__u64	count;
@@ -1079,6 +1089,14 @@ struct xfs_health_monitor_shutdown {
 	__u32	reasons;
 };
 
+/* file range events */
+struct xfs_health_monitor_filerange {
+	__u64	pos;
+	__u64	len;
+	__u64	ino;
+	__u32	gen;
+};
+
 /* disk media errors */
 struct xfs_health_monitor_media {
 	__u64	daddr;
@@ -1107,6 +1125,7 @@ struct xfs_health_monitor_event {
 		struct xfs_health_monitor_inode inode;
 		struct xfs_health_monitor_shutdown shutdown;
 		struct xfs_health_monitor_media media;
+		struct xfs_health_monitor_filerange filerange;
 	} e;
 
 	/* zeroes */
fs/xfs/xfs_healthmon.h +17 -0
index 407c5e1f466726..1ce49197262b1c 100644
@@ -21,6 +21,13 @@ enum xfs_healthmon_type {
 
 	/* media errors */
 	XFS_HEALTHMON_MEDIA_ERROR,
+
+	/* file range events */
+	XFS_HEALTHMON_BUFREAD,
+	XFS_HEALTHMON_BUFWRITE,
+	XFS_HEALTHMON_DIOREAD,
+	XFS_HEALTHMON_DIOWRITE,
+	XFS_HEALTHMON_DATALOST,
 };
 
 enum xfs_healthmon_domain {
@@ -36,6 +43,9 @@ enum xfs_healthmon_domain {
 	XFS_HEALTHMON_DATADEV,
 	XFS_HEALTHMON_RTDEV,
 	XFS_HEALTHMON_LOGDEV,
+
+	/* file range events */
+	XFS_HEALTHMON_FILERANGE,
 };
 
 struct xfs_healthmon_event {
@@ -78,6 +88,13 @@ struct xfs_healthmon_event {
 			xfs_daddr_t	daddr;
 			uint64_t	bbcount;
 		};
+		/* file range events */
+		struct {
+			xfs_ino_t	fino;
+			loff_t		fpos;
+			uint64_t	flen;
+			uint32_t	fgen;
+		};
 	};
 };
 
fs/xfs/xfs_trace.h +59 -0

77 lines not on this page · raw message

fs/xfs/xfs_healthmon.c +96 -2

163 lines not on this page · raw message

fs/xfs/xfs_trace.c +1 -0
index 08ddab700a6cd3..eb35015c091570 100644
@@ -54,6 +54,7 @@
 #include "xfs_health.h"
 #include "xfs_healthmon.h"
 #include "xfs_notify_failure.h"
+#include "xfs_file.h"
 
 /*
  * We include this last to have the helpers above available for the trace
#19 lore raw

PATCH 16/22 xfs: allow reconfiguration of the health monitoring device

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that we can reconfigure the health monitoring device by calling the XFS_IOC_HEALTH_MONITOR ioctl on it.

---
 fs/xfs/xfs_healthmon.c |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
fs/xfs/xfs_healthmon.c +44 -0
index 31c2f6f43cf474..d3784073494ec6 100644
@@ -23,6 +23,7 @@
 #include "xfs_fsops.h"
 #include "xfs_notify_failure.h"
 #include "xfs_file.h"
+#include "xfs_ioctl.h"
 
 #include <linux/anon_inodes.h>
 #include <linux/eventpoll.h>
@@ -1140,6 +1141,48 @@ xfs_healthmon_show_fdinfo(
 }
 #endif
 
+/* Reconfigure the health monitor. */
+STATIC long
+xfs_healthmon_reconfigure(
+	struct file			*file,
+	unsigned int			cmd,
+	void __user			*arg)
+{
+	struct xfs_health_monitor	hmo;
+	struct xfs_healthmon		*hm = file->private_data;
+
+	if (copy_from_user(&hmo, arg, sizeof(hmo)))
+		return -EFAULT;
+
+	if (!xfs_healthmon_validate(&hmo))
+		return -EINVAL;
+
+	mutex_lock(&hm->lock);
+	hm->format = hmo.format;
+	hm->verbose = !!(hmo.flags & XFS_HEALTH_MONITOR_VERBOSE);
+	mutex_unlock(&hm->lock);
+	return 0;
+}
+
+/* Handle ioctls for the health monitoring thread. */
+STATIC long
+xfs_healthmon_ioctl(
+	struct file			*file,
+	unsigned int			cmd,
+	unsigned long			p)
+{
+	void __user			*arg = (void __user *)p;
+
+	switch (cmd) {
+	case XFS_IOC_HEALTH_MONITOR:
+		return xfs_healthmon_reconfigure(file, cmd, arg);
+	default:
+		break;
+	}
+
+	return -ENOTTY;
+}
+
 static const struct file_operations xfs_healthmon_fops = {
 	.owner		= THIS_MODULE,
 #ifdef CONFIG_PROC_FS
@@ -1148,6 +1191,7 @@ static const struct file_operations xfs_healthmon_fops = {
 	.read_iter	= xfs_healthmon_read_iter,
 	.poll		= xfs_healthmon_poll,
 	.release	= xfs_healthmon_release,
+	.unlocked_ioctl	= xfs_healthmon_ioctl,
 };
 
 /*
#20 lore raw

PATCH 17/22 xfs: validate fds against running healthmon

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create a new ioctl for the healthmon file that checks that a given fd points to the same filesystem that the healthmon file is monitoring. This allows xfs_healer to check that when it reopens a mountpoint to perform repairs, the file that it gets matches the filesystem that generated the corruption report.

(Note that xfs_healer doesn't maintain an open fd to a filesystem that it's monitoring so that it doesn't pin the mount.)

---
 fs/xfs/libxfs/xfs_fs.h |   10 ++++++++++
 fs/xfs/xfs_healthmon.c |   32 ++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
fs/xfs/libxfs/xfs_fs.h +10 -0
index a96f11d9bd9c64..2b82535196cdb0 100644
@@ -1147,6 +1147,15 @@ struct xfs_health_monitor {
 /* Initial return format version */
 #define XFS_HEALTH_MONITOR_FMT_V0	(0)
 
+/*
+ * Check that a given fd points to the same filesystem that the health monitor
+ * is monitoring.
+ */
+struct xfs_health_samefs {
+	__s32		fd;
+	__u32		flags;	/* zero for now */
+};
+
 /*
  * ioctl commands that are used by Linux filesystems
  */
@@ -1187,6 +1196,7 @@ struct xfs_health_monitor {
 #define XFS_IOC_SCRUBV_METADATA	_IOWR('X', 64, struct xfs_scrub_vec_head)
 #define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)
 #define XFS_IOC_HEALTH_MONITOR	_IOW ('X', 68, struct xfs_health_monitor)
+#define XFS_IOC_HEALTH_SAMEFS	_IOW ('X', 69, struct xfs_health_samefs)
 
 /*
  * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_healthmon.c +32 -0
index d3784073494ec6..9752b058978995 100644
@@ -1164,6 +1164,36 @@ xfs_healthmon_reconfigure(
 	return 0;
 }
 
+/* Does the fd point to the same filesystem as the one we're monitoring? */
+STATIC long
+xfs_healthmon_samefs(
+	struct file			*file,
+	unsigned int			cmd,
+	void __user			*arg)
+{
+	struct xfs_health_samefs	hms;
+	struct xfs_healthmon		*hm = file->private_data;
+	struct inode			*hms_inode;
+	int				ret = 0;
+
+	if (copy_from_user(&hms, arg, sizeof(hms)))
+		return -EFAULT;
+
+	if (hms.flags)
+		return -EINVAL;
+
+	CLASS(fd, hms_fd)(hms.fd);
+	if (fd_empty(hms_fd))
+		return -EBADF;
+
+	hms_inode = file_inode(fd_file(hms_fd));
+	mutex_lock(&hm->lock);
+	if (!hm->mp || hm->mp->m_super != hms_inode->i_sb)
+		ret = -ESTALE;
+	mutex_unlock(&hm->lock);
+	return ret;
+}
+
 /* Handle ioctls for the health monitoring thread. */
 STATIC long
 xfs_healthmon_ioctl(
@@ -1176,6 +1206,8 @@ xfs_healthmon_ioctl(
 	switch (cmd) {
 	case XFS_IOC_HEALTH_MONITOR:
 		return xfs_healthmon_reconfigure(file, cmd, arg);
+	case XFS_IOC_HEALTH_SAMEFS:
+		return xfs_healthmon_samefs(file, cmd, arg);
 	default:
 		break;
 	}
#21 lore raw

PATCH 18/22 xfs: add media error reporting ioctl

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Add a new privileged ioctl so that xfs_scrub can report media errors to the kernel for further processing.

---
 fs/xfs/libxfs/xfs_fs.h      |   16 ++++
 fs/xfs/xfs_notify_failure.h |    8 ++
 fs/xfs/xfs_trace.h          |    2 
 fs/xfs/Makefile             |    6 -
 fs/xfs/xfs_healthmon.c      |    2 
 fs/xfs/xfs_ioctl.c          |    3 +
 fs/xfs/xfs_notify_failure.c |  187 +++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 213 insertions(+), 11 deletions(-)
fs/xfs/libxfs/xfs_fs.h +16 -0
index 2b82535196cdb0..65fcc94ed9b40c 100644
@@ -1156,6 +1156,21 @@ struct xfs_health_samefs {
 	__u32		flags;	/* zero for now */
 };
 
+/* Report a media error */
+struct xfs_media_error {
+	__u64	flags;		/* flags */
+	__u64	daddr;		/* disk address of range */
+	__u64	bbcount;	/* length, in 512b blocks */
+	__u64	pad;		/* zero */
+};
+
+#define XFS_MEDIA_ERROR_DATADEV	(1)	/* data device */
+#define XFS_MEDIA_ERROR_LOGDEV	(2)	/* external log device */
+#define XFS_MEDIA_ERROR_RTDEV	(3)	/* realtime device */
+
+/* bottom byte of flags is the device code */
+#define XFS_MEDIA_ERROR_DEVMASK	(0xFF)
+
 /*
  * ioctl commands that are used by Linux filesystems
  */
@@ -1197,6 +1212,7 @@ struct xfs_health_samefs {
 #define XFS_IOC_RTGROUP_GEOMETRY _IOWR('X', 65, struct xfs_rtgroup_geometry)
 #define XFS_IOC_HEALTH_MONITOR	_IOW ('X', 68, struct xfs_health_monitor)
 #define XFS_IOC_HEALTH_SAMEFS	_IOW ('X', 69, struct xfs_health_samefs)
+#define XFS_IOC_MEDIA_ERROR	_IOW ('X', 70, struct xfs_media_error)
 
 /*
  * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_notify_failure.h +7 -1
index 2695732ec20875..279f9329a4d5f3 100644
@@ -6,7 +6,9 @@
 #ifndef __XFS_NOTIFY_FAILURE_H__
 #define __XFS_NOTIFY_FAILURE_H__
 
+#if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX)
 extern const struct dax_holder_operations xfs_dax_holder_operations;
+#endif
 
 enum xfs_failed_device {
 	XFS_FAILED_DATADEV,
@@ -14,7 +16,7 @@ enum xfs_failed_device {
 	XFS_FAILED_RTDEV,
 };
 
-#if defined(CONFIG_XFS_LIVE_HOOKS) && defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX)
+#if defined(CONFIG_XFS_LIVE_HOOKS)
 struct xfs_media_error_params {
 	struct xfs_mount		*mp;
 	enum xfs_failed_device		fdev;
@@ -41,4 +43,8 @@ struct xfs_media_error_hook { };
 # define xfs_media_error_hook_setup(...)	((void)0)
 #endif /* CONFIG_XFS_LIVE_HOOKS */
 
+struct xfs_media_error;
+int xfs_ioc_media_error(struct xfs_mount *mp,
+		struct xfs_media_error __user *arg);
+
 #endif /* __XFS_NOTIFY_FAILURE_H__ */
fs/xfs/xfs_trace.h +0 -2
index d1836583d4dfbb..e5d95add53d347 100644
@@ -6213,7 +6213,6 @@ TRACE_EVENT(xfs_healthmon_metadata_hook,
 		  __entry->lost_prev)
 );
 
-#if defined(CONFIG_XFS_LIVE_HOOKS) && defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX)
 TRACE_EVENT(xfs_healthmon_media_error_hook,
 	TP_PROTO(const struct xfs_media_error_params *p,
 		 unsigned int events, unsigned long long lost_prev),
@@ -6261,7 +6260,6 @@ TRACE_EVENT(xfs_healthmon_media_error_hook,
 		  __entry->events,
 		  __entry->lost_prev)
 );
-#endif
 
 #define XFS_FILE_IOERROR_STRINGS \
 	{ XFS_FILE_IOERROR_BUFFERED_READ,	"readahead" }, \
fs/xfs/Makefile +1 -5
index d4e9070a9326ba..2279cb0b874814 100644
@@ -98,6 +98,7 @@ xfs-y				+= xfs_aops.o \
 				   xfs_message.o \
 				   xfs_mount.o \
 				   xfs_mru_cache.o \
+				   xfs_notify_failure.o \
 				   xfs_pwork.o \
 				   xfs_reflink.o \
 				   xfs_stats.o \
@@ -148,11 +149,6 @@ xfs-$(CONFIG_SYSCTL)		+= xfs_sysctl.o
 xfs-$(CONFIG_COMPAT)		+= xfs_ioctl32.o
 xfs-$(CONFIG_EXPORTFS_BLOCK_OPS)	+= xfs_pnfs.o
 
-# notify failure
-ifeq ($(CONFIG_MEMORY_FAILURE),y)
-xfs-$(CONFIG_FS_DAX)		+= xfs_notify_failure.o
-endif
-
 xfs-$(CONFIG_XFS_DRAIN_INTENTS)	+= xfs_drain.o
 xfs-$(CONFIG_XFS_LIVE_HOOKS)	+= xfs_hooks.o
 xfs-$(CONFIG_XFS_MEMORY_BUFS)	+= xfs_buf_mem.o
fs/xfs/xfs_healthmon.c +0 -2
index 9752b058978995..e5715f52f4b218 100644
@@ -523,7 +523,6 @@ xfs_healthmon_shutdown_hook(
 	return NOTIFY_DONE;
 }
 
-#if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_FS_DAX)
 /* Add a media error event to the reporting queue. */
 STATIC int
 xfs_healthmon_media_error_hook(
@@ -574,7 +573,6 @@ xfs_healthmon_media_error_hook(
 	mutex_unlock(&hm->lock);
 	return NOTIFY_DONE;
 }
-#endif
 
 /* Add a file io error event to the reporting queue. */
 STATIC int
fs/xfs/xfs_ioctl.c +3 -0
index 08998d84554f09..7a80a6ad4b2d99 100644
@@ -42,6 +42,7 @@
 #include "xfs_handle.h"
 #include "xfs_rtgroup.h"
 #include "xfs_healthmon.h"
+#include "xfs_notify_failure.h"
 
 #include <linux/mount.h>
 #include <linux/fileattr.h>
@@ -1424,6 +1425,8 @@ xfs_file_ioctl(
 
 	case XFS_IOC_HEALTH_MONITOR:
 		return xfs_ioc_health_monitor(mp, arg);
+	case XFS_IOC_MEDIA_ERROR:
+		return xfs_ioc_media_error(mp, arg);
 
 	default:
 		return -ENOTTY;
fs/xfs/xfs_notify_failure.c +186 -1

198 lines not on this page · raw message

#22 lore raw

PATCH 19/22 xfs: send uevents when major filesystem events happen

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Send uevents when we mount, unmount, and shut down the filesystem, so that we can trigger systemd services when major events happen.

---
 fs/xfs/xfs_super.h |   13 +++++++
 fs/xfs/xfs_fsops.c |   18 ++++++++++
 fs/xfs/xfs_super.c |   94 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 125 insertions(+)
fs/xfs/xfs_super.h +13 -0
index c0e85c1e42f27d..6d428bd04a0248 100644
@@ -101,4 +101,17 @@ extern struct workqueue_struct *xfs_discard_wq;
 
 struct dentry *xfs_debugfs_mkdir(const char *name, struct dentry *parent);
 
+#define XFS_UEVENT_BUFLEN ( \
+	sizeof("SID=") + sizeof_field(struct super_block, s_id) + \
+	sizeof("UUID=") + UUID_STRING_LEN + \
+	sizeof("META_UUID=") + UUID_STRING_LEN)
+
+#define XFS_UEVENT_STR_PTRS \
+	NULL, /* sid */ \
+	NULL, /* uuid */ \
+	NULL /* metauuid */
+
+int xfs_format_uevent_strings(struct xfs_mount *mp, char *buf, ssize_t buflen,
+		char **env);
+
 #endif	/* __XFS_SUPER_H__ */
fs/xfs/xfs_fsops.c +18 -0
index 26ed16e67410d7..0b6b178cb8169a 100644
@@ -522,6 +522,23 @@ xfs_shutdown_hook_setup(
 # define xfs_shutdown_hook(...)		((void)0)
 #endif /* CONFIG_XFS_LIVE_HOOKS */
 
+static void
+xfs_send_shutdown_uevent(
+	struct xfs_mount	*mp)
+{
+	char			buf[XFS_UEVENT_BUFLEN];
+	char			*env[] = {
+		"TYPE=shutdown",
+		XFS_UEVENT_STR_PTRS,
+		NULL,
+	};
+	int			error;
+
+	error = xfs_format_uevent_strings(mp, buf, sizeof(buf), &env[2]);
+	if (!error)
+		kobject_uevent_env(&mp->m_kobj.kobject, KOBJ_OFFLINE, env);
+}
+
 /*
  * Force a shutdown of the filesystem instantly while keeping the filesystem
  * consistent. We don't do an unmount here; just shutdown the shop, make sure
@@ -572,6 +589,7 @@ xfs_do_force_shutdown(
 	}
 
 	trace_xfs_force_shutdown(mp, tag, flags, fname, lnnum);
+	xfs_send_shutdown_uevent(mp);
 
 	xfs_alert_tag(mp, tag,
 "%s (0x%x) detected at %pS (%s:%d).  Shutting down filesystem.",
fs/xfs/xfs_super.c +94 -0

124 lines not on this page · raw message

#23 lore raw

PATCH 20/22 xfs: merge health monitoring events when possible

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Reduce memory consumption and event traffic by merging healthmon events whenever we actually add an event to the queue.

---
 fs/xfs/xfs_trace.h     |    1 
 fs/xfs/xfs_healthmon.c |  107 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
fs/xfs/xfs_trace.h +1 -0
index e5d95add53d347..520526ef9cd11c 100644
@@ -6143,6 +6143,7 @@ DEFINE_EVENT(xfs_healthmon_event_class, name, \
 	TP_PROTO(const struct xfs_mount *mp, const struct xfs_healthmon_event *event), \
 	TP_ARGS(mp, event))
 DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_insert);
+DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_merge);
 DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_push);
 DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_pop);
 DEFINE_HEALTHMONEVENT_EVENT(xfs_healthmon_format);
fs/xfs/xfs_healthmon.c +107 -0

125 lines not on this page · raw message

#24 lore raw

PATCH 21/22 xfs: restrict healthmon users further

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Because health monitoring events include file handles and deep information about the filesystem structure, restrict usage to healthmon to processes that can open the root directory and run in the initial user namespace.

---
 fs/xfs/xfs_healthmon.h |    4 ++--
 fs/xfs/xfs_healthmon.c |    9 ++++++++-
 fs/xfs/xfs_ioctl.c     |    2 +-
 3 files changed, 11 insertions(+), 4 deletions(-)
fs/xfs/xfs_healthmon.h +2 -2
index 1ce49197262b1c..6b650ab0c92238 100644
@@ -99,10 +99,10 @@ struct xfs_healthmon_event {
 };
 
 #ifdef CONFIG_XFS_HEALTH_MONITOR
-long xfs_ioc_health_monitor(struct xfs_mount *mp,
+long xfs_ioc_health_monitor(struct file *file,
 		struct xfs_health_monitor __user *arg);
 #else
-# define xfs_ioc_health_monitor(mp, hmo)	(-ENOTTY)
+# define xfs_ioc_health_monitor(file, hmo)	(-ENOTTY)
 #endif /* CONFIG_XFS_HEALTH_MONITOR */
 
 #endif /* __XFS_HEALTHMON_H__ */
fs/xfs/xfs_healthmon.c +8 -1
index b46b63e62d5143..a8ea6483ca98fb 100644
@@ -1337,18 +1337,25 @@ static const struct file_operations xfs_healthmon_fops = {
  */
 long
 xfs_ioc_health_monitor(
-	struct xfs_mount		*mp,
+	struct file			*file,
 	struct xfs_health_monitor __user *arg)
 {
 	struct xfs_health_monitor	hmo;
 	struct xfs_healthmon		*hm;
 	struct xfs_healthmon_event	*event;
+	struct xfs_inode		*ip = XFS_I(file_inode(file));
+	struct xfs_mount		*mp = ip->i_mount;
 	int				fd;
 	int				ret;
 
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
 
+	if (ip->i_ino != mp->m_sb.sb_rootino)
+		return -EPERM;
+	if (current_user_ns() != &init_user_ns)
+		return -EPERM;
+
 	if (copy_from_user(&hmo, arg, sizeof(hmo)))
 		return -EFAULT;
 
fs/xfs/xfs_ioctl.c +1 -1
index 7a80a6ad4b2d99..6c3eecabf09908 100644
@@ -1424,7 +1424,7 @@ xfs_file_ioctl(
 		return xfs_ioc_commit_range(filp, arg);
 
 	case XFS_IOC_HEALTH_MONITOR:
-		return xfs_ioc_health_monitor(mp, arg);
+		return xfs_ioc_health_monitor(filp, arg);
 	case XFS_IOC_MEDIA_ERROR:
 		return xfs_ioc_media_error(mp, arg);
 
#25 lore raw

PATCH 22/22 xfs: charge healthmon event objects to the memcg of the listening process

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Event objects are created in the context of whichever process experienced the health event, which means that we currently charge that process' memory cgroup controller for that object. This isn't entirely fair to that process, because it's being charged for memory that solely benefits whatever's using the healthmon fd (xfs_healer).

Therefore, save the memcg that was in place when the healthmon fd was created (which we assume is xfs_healer) and make sure the objects are charged to that memcg. This also enables sysadmins to constrain the kernel memory usage of xfs_healer through memcgs.

---
 fs/xfs/xfs_healthmon.c |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
fs/xfs/xfs_healthmon.c +18 -0
index a8ea6483ca98fb..def4de5f6bc543 100644
@@ -67,6 +67,9 @@ struct xfs_healthmon {
 	struct xfs_healthmon_event	*first_event;
 	struct xfs_healthmon_event	*last_event;
 
+	/* charge event object usage to this memory cgroup */
+	struct mem_cgroup		*memcg;
+
 	/* live update hooks */
 	struct xfs_shutdown_hook	shook;
 	struct xfs_health_hook		hhook;
@@ -500,6 +503,7 @@ xfs_healthmon_metadata_hook(
 	struct xfs_health_update_params	*hup = data;
 	struct xfs_healthmon		*hm;
 	struct xfs_healthmon_event	*event;
+	struct mem_cgroup		*old_memcg;
 	enum xfs_health_update_type	type = action;
 	unsigned int			mask = 0;
 	int				error;
@@ -511,6 +515,7 @@ xfs_healthmon_metadata_hook(
 		return NOTIFY_DONE;
 
 	mutex_lock(&hm->lock);
+	old_memcg = set_active_memcg(hm->memcg);
 
 	trace_xfs_healthmon_metadata_hook(hm->mp, action, hup, hm->events,
 			hm->lost_prev_event);
@@ -586,6 +591,7 @@ xfs_healthmon_metadata_hook(
 		goto out_event;
 
 out_unlock:
+	set_active_memcg(old_memcg);
 	mutex_unlock(&hm->lock);
 	return NOTIFY_DONE;
 out_event:
@@ -602,11 +608,13 @@ xfs_healthmon_shutdown_hook(
 {
 	struct xfs_healthmon		*hm;
 	struct xfs_healthmon_event	*event;
+	struct mem_cgroup		*old_memcg;
 	int				error;
 
 	hm = container_of(nb, struct xfs_healthmon, shook.shutdown_hook.nb);
 
 	mutex_lock(&hm->lock);
+	old_memcg = set_active_memcg(hm->memcg);
 
 	trace_xfs_healthmon_shutdown_hook(hm->mp, action, hm->events,
 			hm->lost_prev_event);
@@ -626,6 +634,7 @@ xfs_healthmon_shutdown_hook(
 		kfree(event);
 
 out_unlock:
+	set_active_memcg(old_memcg);
 	mutex_unlock(&hm->lock);
 	return NOTIFY_DONE;
 }
@@ -640,12 +649,14 @@ xfs_healthmon_media_error_hook(
 	struct xfs_healthmon		*hm;
 	struct xfs_healthmon_event	*event;
 	struct xfs_media_error_params	*p = data;
+	struct mem_cgroup		*old_memcg;
 	enum xfs_healthmon_domain	domain = 0; /* shut up gcc */
 	int				error;
 
 	hm = container_of(nb, struct xfs_healthmon, mhook.error_hook.nb);
 
 	mutex_lock(&hm->lock);
+	old_memcg = set_active_memcg(hm->memcg);
 
 	trace_xfs_healthmon_media_error_hook(p, hm->events,
 			hm->lost_prev_event);
@@ -677,6 +688,7 @@ xfs_healthmon_media_error_hook(
 		kfree(event);
 
 out_unlock:
+	set_active_memcg(old_memcg);
 	mutex_unlock(&hm->lock);
 	return NOTIFY_DONE;
 }
@@ -691,6 +703,7 @@ xfs_healthmon_file_ioerror_hook(
 	struct xfs_healthmon		*hm;
 	struct xfs_healthmon_event	*event;
 	struct xfs_file_ioerror_params	*p = data;
+	struct mem_cgroup		*old_memcg;
 	enum xfs_healthmon_type		type = 0;
 	int				error;
 
@@ -709,6 +722,7 @@ xfs_healthmon_file_ioerror_hook(
 	}
 
 	mutex_lock(&hm->lock);
+	old_memcg = set_active_memcg(hm->memcg);
 
 	trace_xfs_healthmon_file_ioerror_hook(hm->mp, action, p, hm->events,
 			hm->lost_prev_event);
@@ -748,6 +762,7 @@ xfs_healthmon_file_ioerror_hook(
 		kfree(event);
 
 out_unlock:
+	set_active_memcg(old_memcg);
 	mutex_unlock(&hm->lock);
 	return NOTIFY_DONE;
 }
@@ -1188,6 +1203,7 @@ xfs_healthmon_release(
 	xfs_healthmon_free_events(hm);
 	if (hm->outbuf.size)
 		kfree(hm->outbuf.buffer);
+	mem_cgroup_put(hm->memcg);
 	kfree(hm);
 
 	return 0;
@@ -1367,6 +1383,7 @@ xfs_ioc_health_monitor(
 		return -ENOMEM;
 	hm->mp = mp;
 	hm->format = hmo.format;
+	hm->memcg = get_mem_cgroup_from_mm(current->mm);
 
 	/*
 	 * Since we already got a ref to the module, take a reference to the
@@ -1443,6 +1460,7 @@ xfs_ioc_health_monitor(
 	xfs_health_hook_disable();
 	mutex_destroy(&hm->lock);
 	xfs_healthmon_free_events(hm);
+	mem_cgroup_put(hm->memcg);
 	kfree(hm);
 	return ret;
 }
#26 lore raw

PATCH v3 2/2 iomap: generic file IO error reporting

In reply to Darrick J. Wong

Hi all,

Refactor the iomap file I/O error handling code so that failures are reported in a generic way to fsnotify. Then connect the XFS health reporting to the same fsnotify, and now XFS can fsnotify userspace of all manner of problems.

This series is much more experimental than the main healer patchset, and I'd rather it not become a blocker for the main patchset. I wouldn't mind rebasing if they went in at the same time though.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=iomap-ioerr-reporting
---
Commits in this patchset:
* iomap: report file IO errors to fsnotify
* xfs: switch healthmon to use the iomap I/O error reporting
* xfs: port notify-failure to use the new vfs io error reporting
* xfs: remove file I/O error hooks
* iomap: remove I/O error hooks
* xfs: report fs metadata errors via fsnotify

---
 fs/xfs/xfs_file.h                 |   37 --------
 fs/xfs/xfs_mount.h                |    3 -
 fs/xfs/xfs_trace.h                |   35 ++++---
 include/linux/fs.h                |   68 ++++++++++++++
 include/linux/iomap.h             |    2 
 Documentation/filesystems/vfs.rst |    7 -
 fs/iomap/buffered-io.c            |    8 +-
 fs/iomap/direct-io.c              |    9 +-
 fs/super.c                        |   53 +++++++++++
 fs/xfs/xfs_aops.c                 |    2 
 fs/xfs/xfs_file.c                 |  174 -------------------------------------
 fs/xfs/xfs_health.c               |   12 ++-
 fs/xfs/xfs_healthmon.c            |   51 ++++++-----
 fs/xfs/xfs_notify_failure.c       |    9 +-
 fs/xfs/xfs_super.c                |    1 
 15 files changed, 190 insertions(+), 281 deletions(-)
#27 lore raw

PATCH 1/6 iomap: report file IO errors to fsnotify

From: Darrick J. Wong <[email protected]>

Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.

---
 include/linux/fs.h     |   64 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/iomap/buffered-io.c |    6 +++++
 fs/iomap/direct-io.c   |    5 ++++
 fs/super.c             |   53 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 128 insertions(+)
include/linux/fs.h +64 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
 	spinlock_t		s_inode_wblist_lock;
 	struct list_head	s_inodes_wb;	/* writeback inodes */
+	struct blocking_notifier_head	s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
 	return true;
 }
 
+enum fs_error_type {
+	/* pagecache reads and writes */
+	FSERR_READAHEAD,
+	FSERR_WRITEBACK,
+
+	/* directio read and writes */
+	FSERR_DIO_READ,
+	FSERR_DIO_WRITE,
+
+	/* media error */
+	FSERR_DATA_LOST,
+
+	/* filesystem metadata */
+	FSERR_METADATA,
+};
+
+struct fs_error {
+	struct work_struct work;
+	struct super_block *sb;
+	struct inode *inode;
+	loff_t pos;
+	u64 len;
+	enum fs_error_type type;
+	int error;
+};
+
+struct fs_error_hook {
+	struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+				struct fs_error_hook *h)
+{
+	return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+				   struct fs_error_hook *h)
+{
+	blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+	h->nb.notifier_call = fn;
+	h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+		enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+	__sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
+			       loff_t pos, u64 len, int error)
+{
+	__sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+
 #endif /* _LINUX_FS_H */
fs/iomap/buffered-io.c +6 -0
index 8dd5421cb910b5..dc19311fe1c6c0 100644
@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
 inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,
 		loff_t pos, u64 len, int error)
 {
+	struct inode *inode = mapping->host;
+
+	inode_error(inode,
+		    direction == READ ? FSERR_READAHEAD : FSERR_WRITEBACK,
+		    pos, len, error);
+
 	if (mapping && mapping->a_ops->ioerror)
 		mapping->a_ops->ioerror(mapping, direction, pos, len,
 				error);
fs/iomap/direct-io.c +5 -0
index 1512d8dbb0d2e7..9f6ce0d9c531bb 100644
@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 
 	if (dops && dops->end_io)
 		ret = dops->end_io(iocb, dio->size, ret, dio->flags);
+	if (dio->error)
+		inode_error(file_inode(iocb->ki_filp),
+			    (dio->flags & IOMAP_DIO_WRITE) ? FSERR_DIO_WRITE :
+							     FSERR_DIO_READ,
+			    offset, dio->size, dio->error);
 	if (dio->error && dops && dops->ioerror)
 		dops->ioerror(file_inode(iocb->ki_filp),
 				(dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
fs/super.c +53 -0

62 lines not on this page · raw message

maintainer Reviewed-by on v1 1/6
#28 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 7 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.

Looks good to me. Feel free to add:

Honza

Quoted 197 lines
---
 include/linux/fs.h     |   64 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/iomap/buffered-io.c |    6 +++++
 fs/iomap/direct-io.c   |    5 ++++
 fs/super.c             |   53 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 128 insertions(+)
include/linux/fs.h +64 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
 	spinlock_t		s_inode_wblist_lock;
 	struct list_head	s_inodes_wb;	/* writeback inodes */
+	struct blocking_notifier_head	s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
 	return true;
 }
 
+enum fs_error_type {
+	/* pagecache reads and writes */
+	FSERR_READAHEAD,
+	FSERR_WRITEBACK,
+
+	/* directio read and writes */
+	FSERR_DIO_READ,
+	FSERR_DIO_WRITE,
+
+	/* media error */
+	FSERR_DATA_LOST,
+
+	/* filesystem metadata */
+	FSERR_METADATA,
+};
+
+struct fs_error {
+	struct work_struct work;
+	struct super_block *sb;
+	struct inode *inode;
+	loff_t pos;
+	u64 len;
+	enum fs_error_type type;
+	int error;
+};
+
+struct fs_error_hook {
+	struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+				struct fs_error_hook *h)
+{
+	return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+				   struct fs_error_hook *h)
+{
+	blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+	h->nb.notifier_call = fn;
+	h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+		enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+	__sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
+			       loff_t pos, u64 len, int error)
+{
+	__sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+
 #endif /* _LINUX_FS_H */
fs/iomap/buffered-io.c +6 -0
index 8dd5421cb910b5..dc19311fe1c6c0 100644
@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
 inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,
 		loff_t pos, u64 len, int error)
 {
+	struct inode *inode = mapping->host;
+
+	inode_error(inode,
+		    direction == READ ? FSERR_READAHEAD : FSERR_WRITEBACK,
+		    pos, len, error);
+
 	if (mapping && mapping->a_ops->ioerror)
 		mapping->a_ops->ioerror(mapping, direction, pos, len,
 				error);
fs/iomap/direct-io.c +5 -0
index 1512d8dbb0d2e7..9f6ce0d9c531bb 100644
@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 
 	if (dops && dops->end_io)
 		ret = dops->end_io(iocb, dio->size, ret, dio->flags);
+	if (dio->error)
+		inode_error(file_inode(iocb->ki_filp),
+			    (dio->flags & IOMAP_DIO_WRITE) ? FSERR_DIO_WRITE :
+							     FSERR_DIO_READ,
+			    offset, dio->size, dio->error);
 	if (dio->error && dops && dops->ioerror)
 		dops->ioerror(file_inode(iocb->ki_filp),
 				(dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
fs/super.c +53 -0
index 5bab94fb7e0358..f6d38e4b3d76b2 100644
@@ -363,6 +363,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
 	spin_lock_init(&s->s_inode_list_lock);
 	INIT_LIST_HEAD(&s->s_inodes_wb);
 	spin_lock_init(&s->s_inode_wblist_lock);
+	BLOCKING_INIT_NOTIFIER_HEAD(&s->s_error_notifier);
 
 	s->s_count = 1;
 	atomic_set(&s->s_active, 1);
@@ -2267,3 +2268,55 @@ int sb_init_dio_done_wq(struct super_block *sb)
 	return 0;
 }
 EXPORT_SYMBOL_GPL(sb_init_dio_done_wq);
+
+static void handle_sb_error(struct work_struct *work)
+{
+	struct fs_error *fserr = container_of(work, struct fs_error, work);
+
+	fsnotify_sb_error(fserr->sb, fserr->inode, fserr->error);
+	blocking_notifier_call_chain(&fserr->sb->s_error_notifier, fserr->type,
+				     fserr);
+	iput(fserr->inode);
+	kfree(fserr);
+}
+
+/**
+ * Report a filesystem error.  The actual work is deferred to a workqueue so
+ * that we're always in process context and to avoid blowing out the caller's
+ * stack.
+ *
+ * @sb Filesystem superblock
+ * @inode Inode within filesystem, if applicable
+ * @type Type of error
+ * @pos Start of file range affected, if applicable
+ * @len Length of file range affected, if applicable
+ * @error Error encountered.
+ */
+void __sb_error(struct super_block *sb, struct inode *inode,
+		enum fs_error_type type, loff_t pos, u64 len, int error)
+{
+	struct fs_error *fserr = kzalloc(sizeof(struct fs_error), GFP_ATOMIC);
+
+	if (!fserr) {
+		printk(KERN_ERR
+ "lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d",
+				inode ? inode->i_ino : 0, type,
+				pos, len, error);
+		return;
+	}
+
+	if (inode) {
+		fserr->sb = inode->i_sb;
+		fserr->inode = igrab(inode);
+	} else {
+		fserr->sb = sb;
+	}
+	fserr->type = type;
+	fserr->pos = pos;
+	fserr->len = len;
+	fserr->error = error;
+	INIT_WORK(&fserr->work, handle_sb_error);
+
+	schedule_work(&fserr->work);
+}
+EXPORT_SYMBOL_GPL(__sb_error);
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#29 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 102 lines · Jan Kara
Quoted 7 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Create a generic hook for iomap filesystems to report IO errors to fsnotify and in-kernel subsystems that want to know about such things.

Looks good to me. Feel free to add:

Honza

Quoted 86 lines
---
 include/linux/fs.h     |   64 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/iomap/buffered-io.c |    6 +++++
 fs/iomap/direct-io.c   |    5 ++++
 fs/super.c             |   53 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 128 insertions(+)
include/linux/fs.h +56 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
      spinlock_t              s_inode_wblist_lock;
      struct list_head        s_inodes_wb;    /* writeback inodes */
+     struct blocking_notifier_head   s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
      return true;
 }
 
+enum fs_error_type {
+     /* pagecache reads and writes */
+     FSERR_READAHEAD,
+     FSERR_WRITEBACK,
+
+     /* directio read and writes */
+     FSERR_DIO_READ,
+     FSERR_DIO_WRITE,
+
+     /* media error */
+     FSERR_DATA_LOST,
+
+     /* filesystem metadata */
+     FSERR_METADATA,
+};
+
+struct fs_error {
+     struct work_struct work;
+     struct super_block *sb;
+     struct inode *inode;
+     loff_t pos;
+     u64 len;
+     enum fs_error_type type;
+     int error;
+};
+
+struct fs_error_hook {
+     struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+                             struct fs_error_hook *h)
+{
+     return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+                                struct fs_error_hook *h)
+{
+     blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+     h->nb.notifier_call = fn;
+     h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+     __sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
Quoted 8 lines

include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135

Quoted 8 lines
include/linux/fs.h +8 -0
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
+                            loff_t pos, u64 len, int error)
+{
+     __sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+

Apart from the fact that Christian is not going to be happy with this bloat of fs.h
shouldn't all this be part of fsnotify.h?

I do not see why ext4 should not use the same workqueue or why any code would need to call fsnotify_sb_error() directly.
...

Quoted 95 lines
Quoted 95 lines
include/linux/fs.h +0 -0
 #endif /* _LINUX_FS_H */
fs/iomap/buffered-io.c +6 -0
index 8dd5421cb910b5..dc19311fe1c6c0 100644
@@ -291,6 +291,12 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
 inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,
              loff_t pos, u64 len, int error)
 {
+     struct inode *inode = mapping->host;
+
+     inode_error(inode,
+                 direction == READ ? FSERR_READAHEAD : FSERR_WRITEBACK,
+                 pos, len, error);
+
      if (mapping && mapping->a_ops->ioerror)
              mapping->a_ops->ioerror(mapping, direction, pos, len,
                              error);
fs/iomap/direct-io.c +5 -0
index 1512d8dbb0d2e7..9f6ce0d9c531bb 100644
@@ -95,6 +95,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 
      if (dops && dops->end_io)
              ret = dops->end_io(iocb, dio->size, ret, dio->flags);
+     if (dio->error)
+             inode_error(file_inode(iocb->ki_filp),
+                         (dio->flags & IOMAP_DIO_WRITE) ? FSERR_DIO_WRITE :
+                                                          FSERR_DIO_READ,
+                         offset, dio->size, dio->error);
      if (dio->error && dops && dops->ioerror)
              dops->ioerror(file_inode(iocb->ki_filp),
                              (dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
fs/super.c +46 -0
index 5bab94fb7e0358..f6d38e4b3d76b2 100644
@@ -363,6 +363,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
      spin_lock_init(&s->s_inode_list_lock);
      INIT_LIST_HEAD(&s->s_inodes_wb);
      spin_lock_init(&s->s_inode_wblist_lock);
+     BLOCKING_INIT_NOTIFIER_HEAD(&s->s_error_notifier);
 
      s->s_count = 1;
      atomic_set(&s->s_active, 1);
@@ -2267,3 +2268,55 @@ int sb_init_dio_done_wq(struct super_block *sb)
      return 0;
 }
 EXPORT_SYMBOL_GPL(sb_init_dio_done_wq);
+
+static void handle_sb_error(struct work_struct *work)
+{
+     struct fs_error *fserr = container_of(work, struct fs_error, work);
+
+     fsnotify_sb_error(fserr->sb, fserr->inode, fserr->error);
+     blocking_notifier_call_chain(&fserr->sb->s_error_notifier, fserr->type,
+                                  fserr);
+     iput(fserr->inode);
+     kfree(fserr);
+}
+
+/**
+ * Report a filesystem error.  The actual work is deferred to a workqueue so
+ * that we're always in process context and to avoid blowing out the caller's
+ * stack.
+ *
+ * @sb Filesystem superblock
+ * @inode Inode within filesystem, if applicable
+ * @type Type of error
+ * @pos Start of file range affected, if applicable
+ * @len Length of file range affected, if applicable
+ * @error Error encountered.
+ */
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error)
+{
+     struct fs_error *fserr = kzalloc(sizeof(struct fs_error), GFP_ATOMIC);
+
+     if (!fserr) {
+             printk(KERN_ERR
+ "lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d",
+                             inode ? inode->i_ino : 0, type,
+                             pos, len, error);
+             return;
+     }
+
+     if (inode) {
+             fserr->sb = inode->i_sb;
+             fserr->inode = igrab(inode);
+     } else {
+             fserr->sb = sb;
+     }
+     fserr->type = type;
+     fserr->pos = pos;
Quoted 8 lines

fs/super.c int sb_init_dio_done_wq(struct super_block *sb) · line 2322

Quoted 8 lines
fs/super.c +7 -0
+     fserr->len = len;
+     fserr->error = error;
+     INIT_WORK(&fserr->work, handle_sb_error);
+
+     schedule_work(&fserr->work);
+}
+EXPORT_SYMBOL_GPL(__sb_error);

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?

If this is needed then we could change fsnotify_sb_error() to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.

Thanks,
Amir.

maintainer
#30 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
      spinlock_t              s_inode_wblist_lock;
      struct list_head        s_inodes_wb;    /* writeback inodes */
+     struct blocking_notifier_head   s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
      return true;
 }
 
+enum fs_error_type {
+     /* pagecache reads and writes */
+     FSERR_READAHEAD,
+     FSERR_WRITEBACK,
+
+     /* directio read and writes */
+     FSERR_DIO_READ,
+     FSERR_DIO_WRITE,
+
+     /* media error */
+     FSERR_DATA_LOST,
+
+     /* filesystem metadata */
+     FSERR_METADATA,
+};
+
+struct fs_error {
+     struct work_struct work;
+     struct super_block *sb;
+     struct inode *inode;
+     loff_t pos;
+     u64 len;
+     enum fs_error_type type;
+     int error;
+};
+
+struct fs_error_hook {
+     struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+                             struct fs_error_hook *h)
+{
+     return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+                                struct fs_error_hook *h)
+{
+     blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+     h->nb.notifier_call = fn;
+     h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+     __sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
Quoted 8 lines
Quoted 5 lines

include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135

Quoted 5 lines
include/linux/fs.h +5 -0
+                            loff_t pos, u64 len, int error)
+{
+     __sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+

Apart from the fact that Christian is not going to be happy with this bloat of fs.h shouldn't all this be part of fsnotify.h?

Point that this maybe doesn't belong to fs.h is a good one. But I don't think fsnotify.h is appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a new fs_error.h header?

Quoted 2 lines

I do not see why ext4 should not use the same workqueue or why any code would need to call fsnotify_sb_error() directly.

Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.

Quoted 35 lines
Quoted 21 lines
Quoted 21 lines
include/linux/fs.h +21 -0
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error)
+{
+     struct fs_error *fserr = kzalloc(sizeof(struct fs_error), GFP_ATOMIC);
+
+     if (!fserr) {
+             printk(KERN_ERR
+ "lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d",
+                             inode ? inode->i_ino : 0, type,
+                             pos, len, error);
+             return;
+     }
+
+     if (inode) {
+             fserr->sb = inode->i_sb;
+             fserr->inode = igrab(inode);
+     } else {
+             fserr->sb = sb;
+     }
+     fserr->type = type;
+     fserr->pos = pos;
Quoted 8 lines

include/linux/fs.h

Quoted 8 lines
include/linux/fs.h +7 -0
+     fserr->len = len;
+     fserr->error = error;
+     INIT_WORK(&fserr->work, handle_sb_error);
+
+     schedule_work(&fserr->work);
+}
+EXPORT_SYMBOL_GPL(__sb_error);

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Quoted 2 lines

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().

Quoted 2 lines

Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?

I think there must be a point of no return during fs shutdown after which we just stop emitting errors.

Quoted 3 lines

If this is needed then we could change fsnotify_sb_error() to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.

This lifetime issue is not limited to fsnotify. I think __sb_error() needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#31 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 92 lines · Jan Kara
Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
      spinlock_t              s_inode_wblist_lock;
      struct list_head        s_inodes_wb;    /* writeback inodes */
+     struct blocking_notifier_head   s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
      return true;
 }
 
+enum fs_error_type {
+     /* pagecache reads and writes */
+     FSERR_READAHEAD,
+     FSERR_WRITEBACK,
+
+     /* directio read and writes */
+     FSERR_DIO_READ,
+     FSERR_DIO_WRITE,
+
+     /* media error */
+     FSERR_DATA_LOST,
+
+     /* filesystem metadata */
+     FSERR_METADATA,
+};
+
+struct fs_error {
+     struct work_struct work;
+     struct super_block *sb;
+     struct inode *inode;
+     loff_t pos;
+     u64 len;
+     enum fs_error_type type;
+     int error;
+};
+
+struct fs_error_hook {
+     struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+                             struct fs_error_hook *h)
+{
+     return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+                                struct fs_error_hook *h)
+{
+     blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+     h->nb.notifier_call = fn;
+     h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+     __sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
Quoted 8 lines
Quoted 5 lines

include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135

Quoted 5 lines
include/linux/fs.h +5 -0
+                            loff_t pos, u64 len, int error)
+{
+     __sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+

Apart from the fact that Christian is not going to be happy with this bloat of fs.h shouldn't all this be part of fsnotify.h?

Quoted 4 lines

Point that this maybe doesn't belong to fs.h is a good one. But I don't think fsnotify.h is appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a new fs_error.h header?

Fine with me, though it'd be a small header.

Quoted 5 lines
Quoted 2 lines

I do not see why ext4 should not use the same workqueue or why any code would need to call fsnotify_sb_error() directly.

Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.

Yes, that should be a trivial patch to change fsnotify_sb_error -> sb_error/inode_error.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
include/linux/fs.h +26 -0
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error)
+{
+     struct fs_error *fserr = kzalloc(sizeof(struct fs_error), GFP_ATOMIC);
+
+     if (!fserr) {
+             printk(KERN_ERR
+ "lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d",
+                             inode ? inode->i_ino : 0, type,
+                             pos, len, error);
+             return;
+     }
+
+     if (inode) {
+             fserr->sb = inode->i_sb;
+             fserr->inode = igrab(inode);
+     } else {
+             fserr->sb = sb;
+     }
+     fserr->type = type;
+     fserr->pos = pos;
+     fserr->len = len;
+     fserr->error = error;
+     INIT_WORK(&fserr->work, handle_sb_error);
+
+     schedule_work(&fserr->work);
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

include/linux/fs.h

Quoted 3 lines
include/linux/fs.h +2 -0
+}
+EXPORT_SYMBOL_GPL(__sb_error);

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...

Quoted 7 lines
Quoted 2 lines

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().

I wonder, could __sb_error call get_active_super() to obtain an active reference to the sb, and then deactivate_super() it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.

The igrab/iput was supposed to prevent the same UAF from happening with the inode, but I should've checked for a non-null return value.

Quoted 5 lines
Quoted 2 lines

Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?

I think there must be a point of no return during fs shutdown after which we just stop emitting errors.

I agree, once S_ACTIVE hits zero there's no point in sending further errors.

Quoted 3 lines
Quoted 3 lines

If this is needed then we could change fsnotify_sb_error() to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.

That would be another way to do it. The sole downstream consumer of the s_error_notifier-based events only cares about ino/gen.

Quoted 4 lines

This lifetime issue is not limited to fsnotify. I think __sb_error() needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).

<nod>

--D

Quoted 5 lines

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#32 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 156 lines · Darrick J. Wong
Quoted 92 lines · Jan Kara
Quoted 82 lines · Amir Goldstein
Quoted 81 lines · Jan Kara
Quoted 81 lines
include/linux/fs.h +59 -0
index 5e4b3a4b24823f..1cb3965db3275c 100644
@@ -80,6 +80,7 @@ struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
 struct iomap_ops;
+struct notifier_head;
 
 extern void __init inode_init(void);
 extern void __init inode_init_early(void);
@@ -1587,6 +1588,7 @@ struct super_block {
 
      spinlock_t              s_inode_wblist_lock;
      struct list_head        s_inodes_wb;    /* writeback inodes */
+     struct blocking_notifier_head   s_error_notifier;
 } __randomize_layout;
 
 static inline struct user_namespace *i_user_ns(const struct inode *inode)
@@ -4069,4 +4071,66 @@ static inline bool extensible_ioctl_valid(unsigned int cmd_a,
      return true;
 }
 
+enum fs_error_type {
+     /* pagecache reads and writes */
+     FSERR_READAHEAD,
+     FSERR_WRITEBACK,
+
+     /* directio read and writes */
+     FSERR_DIO_READ,
+     FSERR_DIO_WRITE,
+
+     /* media error */
+     FSERR_DATA_LOST,
+
+     /* filesystem metadata */
+     FSERR_METADATA,
+};
+
+struct fs_error {
+     struct work_struct work;
+     struct super_block *sb;
+     struct inode *inode;
+     loff_t pos;
+     u64 len;
+     enum fs_error_type type;
+     int error;
+};
+
+struct fs_error_hook {
+     struct notifier_block nb;
+};
+
+static inline int sb_hook_error(struct super_block *sb,
+                             struct fs_error_hook *h)
+{
+     return blocking_notifier_chain_register(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_unhook_error(struct super_block *sb,
+                                struct fs_error_hook *h)
+{
+     blocking_notifier_chain_unregister(&sb->s_error_notifier, &h->nb);
+}
+
+static inline void sb_init_error_hook(struct fs_error_hook *h, notifier_fn_t fn)
+{
+     h->nb.notifier_call = fn;
+     h->nb.priority = 0;
+}
+
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error);
+
+static inline void sb_error(struct super_block *sb, int error)
+{
+     __sb_error(sb, NULL, FSERR_METADATA, 0, 0, error);
+}
+
+static inline void inode_error(struct inode *inode, enum fs_error_type type,
Quoted 8 lines
Quoted 5 lines

include/linux/fs.h static inline bool extensible_ioctl_valid(unsigned int cmd_a, · line 4135

Quoted 5 lines
include/linux/fs.h +5 -0
+                            loff_t pos, u64 len, int error)
+{
+     __sb_error(inode->i_sb, inode, type, pos, len, error);
+}
+

Apart from the fact that Christian is not going to be happy with this bloat of fs.h shouldn't all this be part of fsnotify.h?

Quoted 4 lines

Point that this maybe doesn't belong to fs.h is a good one. But I don't think fsnotify.h is appropriate either because this isn't really part of fsnotify. It is a layer on top that's binding fsnotify and notifier chain notification. So maybe a new fs_error.h header?

Fine with me, though it'd be a small header.

Quoted 5 lines
Quoted 2 lines

I do not see why ext4 should not use the same workqueue or why any code would need to call fsnotify_sb_error() directly.

Yes, I guess we can convert ext4 to the same framework but I'm fine with cleaning that up later.

Yes, that should be a trivial patch to change fsnotify_sb_error -> sb_error/inode_error.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
include/linux/fs.h +26 -0
+void __sb_error(struct super_block *sb, struct inode *inode,
+             enum fs_error_type type, loff_t pos, u64 len, int error)
+{
+     struct fs_error *fserr = kzalloc(sizeof(struct fs_error), GFP_ATOMIC);
+
+     if (!fserr) {
+             printk(KERN_ERR
+ "lost fs error report for ino %lu type %u pos 0x%llx len 0x%llx error %d",
+                             inode ? inode->i_ino : 0, type,
+                             pos, len, error);
+             return;
+     }
+
+     if (inode) {
+             fserr->sb = inode->i_sb;
+             fserr->inode = igrab(inode);
+     } else {
+             fserr->sb = sb;
+     }
+     fserr->type = type;
+     fserr->pos = pos;
+     fserr->len = len;
+     fserr->error = error;
+     INIT_WORK(&fserr->work, handle_sb_error);
+
+     schedule_work(&fserr->work);
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

include/linux/fs.h

Quoted 3 lines
include/linux/fs.h +2 -0
+}
+EXPORT_SYMBOL_GPL(__sb_error);

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...

Quoted 7 lines
Quoted 2 lines

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().

Quoted 4 lines

I wonder, could __sb_error call get_active_super() to obtain an active reference to the sb, and then deactivate_super() it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.

...and now that I've actually tried it, I realize that we can't actually call get_active_super because it can sleep waiting for s_umount and SB_BORN. Maybe we could directly atomic_inc_not_zero(&sb->s_active) adn trust that the caller has an active ref to the sb? I think that's true for anyone calling __sb_error with a non-null inode.

--D

Quoted 34 lines

The igrab/iput was supposed to prevent the same UAF from happening with the inode, but I should've checked for a non-null return value.

Quoted 5 lines
Quoted 2 lines

Another question to ask is whether reporting fs error duing fs shutdown is a feature or anti feature?

I think there must be a point of no return during fs shutdown after which we just stop emitting errors.

I agree, once S_ACTIVE hits zero there's no point in sending further errors.

Quoted 3 lines
Quoted 3 lines

If this is needed then we could change fsnotify_sb_error() to take ino,gen or file handle directly instead of calling filesystem to encode a file handle to report with the event.

That would be another way to do it. The sole downstream consumer of the s_error_notifier-based events only cares about ino/gen.

Quoted 4 lines

This lifetime issue is not limited to fsnotify. I think __sb_error() needs to check whether the superblock is still alive and synchronize properly with sb shutdown (at which point making ext4 use this framework will be a net win because it will close this race for ext4 as well).

<nod>

--D

Quoted 5 lines

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#33 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 26 lines · Darrick J. Wong
Quoted 20 lines · Darrick J. Wong
Quoted 6 lines · Jan Kara
Quoted 5 lines · Amir Goldstein

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...

Quoted 7 lines
Quoted 2 lines

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().

Quoted 4 lines

I wonder, could __sb_error call get_active_super() to obtain an active reference to the sb, and then deactivate_super() it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.

Quoted 5 lines

...and now that I've actually tried it, I realize that we can't actually call get_active_super because it can sleep waiting for s_umount and SB_BORN. Maybe we could directly atomic_inc_not_zero(&sb->s_active) adn trust that the caller has an active ref to the sb? I think that's true for anyone calling __sb_error with a non-null inode.

Well, the side-effects of holding active sb reference from some workqueue item tend to hit back occasionally (like when userspace assumes the device isn't used anymore but it in fact still is because of the active reference). Every time we tried something like this (last time it was with iouring I believe) some user came back and complained his setup broke. In this case it should be really rare but still I think it's better to avoid it if we can (plus I'm not sure what you'd like to do for __sb_error() callers that don't get the inode and thus active reference isn't really guaranteed - they still need the protection against umount so that handle_sb_error() can do the notifier callchain thing).

So I think a better solution might be that generic_shutdown_super() waits for pending error notifications after clearing SB_ACTIVE before umount proceeds further and __sb_error() just starts discarding new notifications as soon as we see SB_ACTIVE is clear.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#34 lore raw

Re: [PATCH 1/6] iomap: report file IO errors to fsnotify

Quoted 44 lines · Jan Kara
Quoted 26 lines · Darrick J. Wong
Quoted 20 lines · Darrick J. Wong
Quoted 6 lines · Jan Kara
Quoted 5 lines · Amir Goldstein

...
We recently discovered that fsnotify_sb_error() calls are exposed to races with generic_shutdown_super():
https://lore.kernel.org/linux-fsdevel/scmyycf2trich22v25s6gpe3ib6ejawflwf76znxg7sedqablp@ejfycd34xvpa/

Hrmm. I've noticed that ever since I added this new patchset, I've been getting more instances of outright crashes in the timer code, or workqueue lockups. I wonder if that UAF is what's going on here...

Quoted 7 lines
Quoted 2 lines

Will punting all FS_ERROR events to workqueue help to improve this situation or will it make it worse?

Worse. But you raise a really good point which I've missed during my review. Currently there's nothing which synchronizes pending works with superblock getting destroyed with obvious UAF issues already in handle_sb_error().

Quoted 4 lines

I wonder, could __sb_error call get_active_super() to obtain an active reference to the sb, and then deactivate_super() it in the workqueue callback? If we can't get an active ref then we presume that the fs is already shutting down and don't send the event.

Quoted 5 lines

...and now that I've actually tried it, I realize that we can't actually call get_active_super because it can sleep waiting for s_umount and SB_BORN. Maybe we could directly atomic_inc_not_zero(&sb->s_active) adn trust that the caller has an active ref to the sb? I think that's true for anyone calling __sb_error with a non-null inode.

Well, the side-effects of holding active sb reference from some workqueue item tend to hit back occasionally (like when userspace assumes the device isn't used anymore but it in fact still is because of the active reference). Every time we tried something like this (last time it was with iouring I believe) some user came back and complained his setup broke. In this case it should be really rare but still I think it's better to avoid it if we can (plus I'm not sure what you'd like to do for __sb_error() callers that don't get the inode and thus active reference isn't really guaranteed - they still need the protection against umount so that handle_sb_error() can do the notifier callchain thing).

Quoted 4 lines

So I think a better solution might be that generic_shutdown_super() waits for pending error notifications after clearing SB_ACTIVE before umount proceeds further and __sb_error() just starts discarding new notifications as soon as we see SB_ACTIVE is clear.

<nod> Summarizing what we just talked about on the ext4 call--

Instead of grabbing an active reference to the sb, I'll instead fix __sb_error to (a) ignore !BORN || !ACTIVE || DYING supers, and (b) increment a counter in the sb whenever we queue an event, and decrement it when the worker finishes with it. generic_shutdown_super can then wait (having just cleared ACTIVE) for the counter to hit zero before it stops fsnotify and drops the dentry cache.

Or at least that's what I'll try today.

--D

Quoted 5 lines

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#35 lore raw

PATCH 2/6 xfs: switch healthmon to use the iomap I/O error reporting

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Use the new generic I/O error reporting paths so that we can remove the xfs-specific hooks.

---
 fs/xfs/xfs_trace.h     |   35 ++++++++++++++++++---------------
 fs/xfs/xfs_healthmon.c |   51 +++++++++++++++++++++++++-----------------------
 2 files changed, 46 insertions(+), 40 deletions(-)
fs/xfs/xfs_trace.h +19 -16
index 520526ef9cd11c..bb7335b56a53e4 100644
@@ -6262,24 +6262,25 @@ TRACE_EVENT(xfs_healthmon_media_error_hook,
 		  __entry->lost_prev)
 );
 
-#define XFS_FILE_IOERROR_STRINGS \
-	{ XFS_FILE_IOERROR_BUFFERED_READ,	"readahead" }, \
-	{ XFS_FILE_IOERROR_BUFFERED_WRITE,	"writeback" }, \
-	{ XFS_FILE_IOERROR_DIRECT_READ,		"directio_read" }, \
-	{ XFS_FILE_IOERROR_DIRECT_WRITE,	"directio_write" }, \
-	{ XFS_FILE_IOERROR_DATA_LOST,		"datalost" }
+#define FS_ERROR_STRINGS \
+	{ FSERR_READAHEAD,	"readahead" }, \
+	{ FSERR_WRITEBACK,	"writeback" }, \
+	{ FSERR_DIO_READ,	"directio_read" }, \
+	{ FSERR_DIO_WRITE,	"directio_write" }, \
+	{ FSERR_DATA_LOST,	"datalost" }, \
+	{ FSERR_METADATA,	"metadata" }
 
-
-TRACE_DEFINE_ENUM(XFS_FILE_IOERROR_BUFFERED_READ);
-TRACE_DEFINE_ENUM(XFS_FILE_IOERROR_BUFFERED_WRITE);
-TRACE_DEFINE_ENUM(XFS_FILE_IOERROR_DIRECT_READ);
-TRACE_DEFINE_ENUM(XFS_FILE_IOERROR_DIRECT_WRITE);
-TRACE_DEFINE_ENUM(XFS_FILE_IOERROR_DATA_LOST);
+TRACE_DEFINE_ENUM(FSERR_READAHEAD);
+TRACE_DEFINE_ENUM(FSERR_WRITEBACK);
+TRACE_DEFINE_ENUM(FSERR_DIO_READ);
+TRACE_DEFINE_ENUM(FSERR_DIO_WRITE);
+TRACE_DEFINE_ENUM(FSERR_DATA_LOST);
+TRACE_DEFINE_ENUM(FSERR_METADATA);
 
 TRACE_EVENT(xfs_healthmon_file_ioerror_hook,
 	TP_PROTO(const struct xfs_mount *mp,
 		 unsigned long action,
-		 const struct xfs_file_ioerror_params *p,
+		 const struct fs_error *p,
 		 unsigned int events, unsigned long long lost_prev),
 	TP_ARGS(mp, action, p, events, lost_prev),
 	TP_STRUCT__entry(
@@ -6294,10 +6295,12 @@ TRACE_EVENT(xfs_healthmon_file_ioerror_hook,
 		__field(unsigned long long, lost_prev)
 	),
 	TP_fast_assign(
+		struct xfs_inode *ip = XFS_I(p->inode);
+
 		__entry->dev = mp ? mp->m_super->s_dev : 0;
 		__entry->action = action;
-		__entry->ino = p->ino;
-		__entry->gen = p->gen;
+		__entry->ino = ip->i_ino;
+		__entry->gen = p->inode->i_generation;
 		__entry->pos = p->pos;
 		__entry->len = p->len;
 		__entry->events = events;
@@ -6307,7 +6310,7 @@ TRACE_EVENT(xfs_healthmon_file_ioerror_hook,
 		  MAJOR(__entry->dev), MINOR(__entry->dev),
 		  __entry->ino,
 		  __entry->gen,
-		  __print_symbolic(__entry->action, XFS_FILE_IOERROR_STRINGS),
+		  __print_symbolic(__entry->action, FS_ERROR_STRINGS),
 		  __entry->pos,
 		  __entry->len,
 		  __entry->events,
fs/xfs/xfs_healthmon.c +27 -24

109 lines not on this page · raw message

#36 lore raw

PATCH 3/6 xfs: port notify-failure to use the new vfs io error reporting

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Port the media error notification code to use the new generic reporting code.

---
 fs/xfs/xfs_notify_failure.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
fs/xfs/xfs_notify_failure.c +5 -4
index bf6e1865d5c3a5..20f040a35537f6 100644
@@ -178,9 +178,10 @@ xfs_dax_failure_fn(
 		invalidate_inode_pages2_range(mapping, pgoff,
 					      pgoff + pgcnt - 1);
 
-	xfs_inode_media_error(ip,
+	inode_error(VFS_I(ip), FSERR_DATA_LOST,
 			XFS_FSB_TO_B(mp, (u64)pgoff << PAGE_SHIFT),
-			XFS_FSB_TO_B(mp, (u64)pgcnt << PAGE_SHIFT));
+			XFS_FSB_TO_B(mp, (u64)pgcnt << PAGE_SHIFT),
+			-EIO);
 
 	xfs_irele(ip);
 	return error;
@@ -496,8 +497,8 @@ xfs_report_one_data_lost(
 	if (rmap_end > lost_end)
 		blocks -= rmap_end - lost_end;
 
-	xfs_inode_media_error(ip, XFS_FSB_TO_B(mp, fileoff),
-			XFS_FSB_TO_B(mp, blocks));
+	inode_error(VFS_I(ip), FSERR_DATA_LOST, XFS_FSB_TO_B(mp, fileoff),
+			XFS_FSB_TO_B(mp, blocks), -EIO);
 
 	xfs_irele(ip);
 	return 0;
#37 lore raw

PATCH 4/6 xfs: remove file I/O error hooks

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Remove these hooks since iomap now does that on its own.

---
 fs/xfs/xfs_file.h  |   37 -----------
 fs/xfs/xfs_mount.h |    3 -
 fs/xfs/xfs_aops.c  |    2 -
 fs/xfs/xfs_file.c  |  174 ----------------------------------------------------
 fs/xfs/xfs_super.c |    1 
 5 files changed, 1 insertion(+), 216 deletions(-)
fs/xfs/xfs_file.h +0 -37
index 441f8a693bb884..2ad91f755caf35 100644
@@ -12,41 +12,4 @@ extern const struct file_operations xfs_dir_file_operations;
 bool xfs_is_falloc_aligned(struct xfs_inode *ip, loff_t pos,
 		long long int len);
 
-enum xfs_file_ioerror_type {
-	XFS_FILE_IOERROR_BUFFERED_READ,
-	XFS_FILE_IOERROR_BUFFERED_WRITE,
-	XFS_FILE_IOERROR_DIRECT_READ,
-	XFS_FILE_IOERROR_DIRECT_WRITE,
-	XFS_FILE_IOERROR_DATA_LOST,
-};
-
-struct xfs_file_ioerror_params {
-	xfs_ino_t		ino;
-	loff_t			pos;
-	u64			len;
-	u32			gen;
-	int			error;
-};
-
-#ifdef CONFIG_XFS_LIVE_HOOKS
-struct xfs_file_ioerror_hook {
-	struct xfs_hook			ioerror_hook;
-};
-
-int xfs_file_ioerror_hook_add(struct xfs_mount *mp,
-		struct xfs_file_ioerror_hook *hook);
-void xfs_file_ioerror_hook_del(struct xfs_mount *mp,
-		struct xfs_file_ioerror_hook *hook);
-void xfs_file_ioerror_hook_setup(struct xfs_file_ioerror_hook *hook,
-		notifier_fn_t mod_fn);
-
-void xfs_vm_ioerror(struct address_space *mapping, int direction, loff_t pos,
-		u64 len, int error);
-
-void xfs_inode_media_error(struct xfs_inode *ip, loff_t pos, u64 len);
-#else
-# define xfs_vm_ioerror			NULL
-# define xfs_inode_media_error(...)	((void)0)
-#endif /* CONFIG_XFS_LIVE_HOOKS */
-
 #endif /* __XFS_FILE_H__ */
fs/xfs/xfs_mount.h +0 -3
index 2d7f9ccba5287e..0feb0fb685f51f 100644
@@ -352,9 +352,6 @@ typedef struct xfs_mount {
 
 	/* Hook to feed media error events to a daemon. */
 	struct xfs_hooks	m_media_error_hooks;
-
-	/* Hook to feed file io error events to a daemon. */
-	struct xfs_hooks	m_file_ioerror_hooks;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_aops.c +0 -2
index f3f28b9ae0f70e..a26f798155331f 100644
@@ -22,7 +22,6 @@
 #include "xfs_icache.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_rtgroup.h"
-#include "xfs_file.h"
 
 struct xfs_writepage_ctx {
 	struct iomap_writepage_ctx ctx;
@@ -811,7 +810,6 @@ const struct address_space_operations xfs_address_space_operations = {
 	.is_partially_uptodate  = iomap_is_partially_uptodate,
 	.error_remove_folio	= generic_error_remove_folio,
 	.swap_activate		= xfs_vm_swap_activate,
-	.ioerror		= xfs_vm_ioerror,
 };
 
 const struct address_space_operations xfs_dax_aops = {
fs/xfs/xfs_file.c +1 -173

192 lines not on this page · raw message

fs/xfs/xfs_super.c +0 -1
index bfd12ccaa707a8..4a8d439ff57408 100644
@@ -2388,7 +2388,6 @@ xfs_init_fs_context(
 	xfs_hooks_init(&mp->m_shutdown_hooks);
 	xfs_hooks_init(&mp->m_health_update_hooks);
 	xfs_hooks_init(&mp->m_media_error_hooks);
-	xfs_hooks_init(&mp->m_file_ioerror_hooks);
 
 	fc->s_fs_info = mp;
 	fc->ops = &xfs_context_ops;
#38 lore raw

PATCH 5/6 iomap: remove I/O error hooks

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Remove the I/O error hooks from struct address_space and iomap_dio_ops because there are no more callers.

---
 include/linux/fs.h                |    4 ----
 include/linux/iomap.h             |    2 --
 Documentation/filesystems/vfs.rst |    7 -------
 fs/iomap/buffered-io.c            |    4 ----
 fs/iomap/direct-io.c              |    4 ----
 5 files changed, 21 deletions(-)
include/linux/fs.h +0 -4
index 1cb3965db3275c..6e3a7cbefbca8a 100644
@@ -478,10 +478,6 @@ struct address_space_operations {
 				sector_t *span);
 	void (*swap_deactivate)(struct file *file);
 	int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
-
-	/* Callback for dealing with IO errors during readahead or writeback */
-	void (*ioerror)(struct address_space *mapping, int direction,
-			loff_t pos, u64 len, int error);
 };
 
 extern const struct address_space_operations empty_aops;
include/linux/iomap.h +0 -2
index ca1590e5002342..73dceabc21c8c7 100644
@@ -486,8 +486,6 @@ struct iomap_dio_ops {
 		      unsigned flags);
 	void (*submit_io)(const struct iomap_iter *iter, struct bio *bio,
 		          loff_t file_offset);
-	void (*ioerror)(struct inode *inode, int direction, loff_t pos,
-			u64 len, int error);
 
 	/*
 	 * Filesystems wishing to attach private information to a direct io bio
Documentation/filesystems/vfs.rst +0 -7
index 9e70006bf99a63..4f13b01e42eb5e 100644
@@ -822,8 +822,6 @@ cache in your filesystem.  The following members are defined:
 		int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span)
 		int (*swap_deactivate)(struct file *);
 		int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
-		void (*ioerror)(struct address_space *mapping, int direction,
-				loff_t pos, u64 len, int error);
 	};
 
 ``read_folio``
@@ -1034,11 +1032,6 @@ cache in your filesystem.  The following members are defined:
 ``swap_rw``
 	Called to read or write swap pages when SWP_FS_OPS is set.
 
-``ioerror``
-        Called to deal with IO errors during readahead or writeback.
-        This may be called from interrupt context, and without any
-        locks necessarily being held.
-
 The File Object
 ===============
 
fs/iomap/buffered-io.c +0 -4
index dc19311fe1c6c0..32628550093f65 100644
@@ -296,10 +296,6 @@ inline void iomap_mapping_ioerror(struct address_space *mapping, int direction,
 	inode_error(inode,
 		    direction == READ ? FSERR_READAHEAD : FSERR_WRITEBACK,
 		    pos, len, error);
-
-	if (mapping && mapping->a_ops->ioerror)
-		mapping->a_ops->ioerror(mapping, direction, pos, len,
-				error);
 }
 
 /**
fs/iomap/direct-io.c +0 -4
index 9f6ce0d9c531bb..1f140031416a0c 100644
@@ -100,10 +100,6 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 			    (dio->flags & IOMAP_DIO_WRITE) ? FSERR_DIO_WRITE :
 							     FSERR_DIO_READ,
 			    offset, dio->size, dio->error);
-	if (dio->error && dops && dops->ioerror)
-		dops->ioerror(file_inode(iocb->ki_filp),
-				(dio->flags & IOMAP_DIO_WRITE) ? WRITE : READ,
-				offset, dio->size, dio->error);
 
 	if (likely(!ret)) {
 		ret = dio->size;
#39 lore raw

PATCH 6/6 xfs: report fs metadata errors via fsnotify

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Report filesystem corruption problems to fsnotify.

---
 fs/xfs/xfs_health.c |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
fs/xfs/xfs_health.c +11 -1
index da827060853a8f..0da4ae216dc169 100644
@@ -71,6 +71,9 @@ xfs_fs_health_update_hook(
 	unsigned int			old_mask,
 	unsigned int			new_mask)
 {
+	if (op != XFS_HEALTHUP_HEALTHY && new_mask)
+		sb_error(mp->m_super, -EFSCORRUPTED);
+
 	if (xfs_hooks_switched_on(&xfs_health_hooks_switch)) {
 		struct xfs_health_update_params	p = {
 			.domain		= XFS_HEALTHUP_FS,
@@ -91,13 +94,17 @@ xfs_group_health_update_hook(
 	unsigned int			old_mask,
 	unsigned int			new_mask)
 {
+	struct xfs_mount		*mp = xg->xg_mount;
+
+	if (op != XFS_HEALTHUP_HEALTHY && new_mask)
+		sb_error(mp->m_super, -EFSCORRUPTED);
+
 	if (xfs_hooks_switched_on(&xfs_health_hooks_switch)) {
 		struct xfs_health_update_params	p = {
 			.old_mask	= old_mask,
 			.new_mask	= new_mask,
 			.group		= xg->xg_gno,
 		};
-		struct xfs_mount	*mp = xg->xg_mount;
 
 		switch (xg->xg_type) {
 		case XG_TYPE_AG:
@@ -124,6 +131,9 @@ xfs_inode_health_update_hook(
 	unsigned int			old_mask,
 	unsigned int			new_mask)
 {
+	if (op != XFS_HEALTHUP_HEALTHY && new_mask)
+		inode_error(VFS_I(ip), FSERR_METADATA, 0, 0, -EFSCORRUPTED);
+
 	if (xfs_hooks_switched_on(&xfs_health_hooks_switch)) {
 		struct xfs_health_update_params	p = {
 			.domain		= XFS_HEALTHUP_INODE,