Skip to contentSkip to conversation

xfs: all pending online scrub improvements

PATCH In review Darrick J. Wong 5 participants 216 messages Dec 30, 2022 – Mar 7, 2023 Page 1 of 5

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/4 xfs: all pending online scrub improvements

Hi everyone,

As I've mentioned several times throughout 2022, I would like to merge the online fsck feature in time for the 2023 LTS kernel. The first big step in this process is to merge all the pending bug fixes, validation improvements, and general reorganization of the existing metadata scrubbing functionality.

This first deluge starts with the design document for the entirety of the online fsck feature. The design doc should be familiar to most of you, as it's been on the list for review for months already. It outlines in brief the problems we're trying to solve, the use cases and testing plan, and the fundamental data structures and algorithms underlying the entire feature.

After that come all the code changes to wrap up the metadata checking part of the feature. The biggest piece here is the scrub drains that allow scrub to quiesce deferred ops targeting AGs so that it can cross-reference recordsets. Most of the rest is tweaking the btree code so that we can do keyspace scans to look for conflicting records.

For this review, I would like people to focus the following:

- Are the major subsystems sufficiently documented that you could figure out what the code does?

- Do you see any problems that are severe enough to cause long term support hassles? (e.g. bad API design, writing weird metadata to disk)

- Can you spot mis-interactions between the subsystems?

- What were my blind spots in devising this feature?

- Are there missing pieces that you'd like to help build?

- Can I just merge all of this?

The one thing that is /not/ in scope for this review are requests for more refactoring of existing subsystems. While there are usually valid arguments for performing such cleanups, those are separate tasks to be prioritized separately. I will get to them after merging online fsck.

I've been running daily online scrubs of every computer I own for the last five years, which has helped me iron out real problems in (limited scope) production. All issues observed in that time have been corrected in this submission.

As a warning, the patches will likely take several days to trickle in. All four patch deluges are based off kernel 6.2-rc1, xfsprogs 6.1, and fstests 2022-12-25.

Thank you all for your participation in the XFS community. Have a safe New Years, and I'll see you all next year!

--D

#1 lore raw

cover xfs: design documentation for online fsck

Hi all,

After six years of development and a nearly two year hiatus from patchbombing, I think it is time to resume the process of merging the online fsck feature into XFS. The full patchset comprises 105 separate patchsets that capture 470 patches across the kernel, xfsprogs, and fstests projects.

I would like to merge this feature into upstream in time for the 2023 LTS kernel. As of 5.15 (aka last year's LTS), we have merged all generally useful infrastructure improvements into the regular filesystem. The only changes to the core filesystem that remain are the ones that are only useful to online fsck itself. In other words, the vast majority of the new code in the patchsets comprising the online fsck feature are is mostly self contained and can be turned off via Kconfig.

Many of you readers might be wondering -- why have I chosen to make one large submission with 100+ patchsets comprising ~500 patches? Why didn't I merge small pieces of functionality bit by bit and revise common code as necessary? Well, the simple answer is that in the past six years, the fundamental algorithms have been revised repeatedly as I've built out the functionality. In other words, the codebase as it is now has the benefit that I now know every piece that's necessary to get the job done in a reasonable manner and within the constraints laid out by community reviews. I believe this has reduced code churn in mainline and freed up my time so that I can iterate faster.

As a concession to the mail servers, I'm breaking up the submission into smaller pieces; I'm only pushing the design document and the revisions to the existing scrub code, which is the first 20% of the patches. Also, I'm arbitrarily restarting the version numbering by reversioning all patchsets from version 22 to epoch 23, version 1.

The big question to everyone reading this is: How might I convince you that there is more merit in merging the whole feature and dealing with the consequences than continuing to maintain it out of tree?

---------

To prepare the XFS community and potential patch reviewers for the upstream submission of the online fsck feature, I decided to write a document capturing the broader picture behind the online repair development effort. The document begins by defining the problems that online fsck aims to solve and outlining specific use cases for the functionality.

Using that as a base, the rest of the design document presents the high level algorithms that fulfill the goals set out at the start and the interactions between the large pieces of the system. Case studies round out the design documentation by adding the details of exactly how specific parts of the online fsck code integrate the algorithms with the filesystem.

The goal of this effort is to help the XFS community understand how the gigantic online repair patchset works. The questions I submit to the community reviewers are:

1. As you read the design doc (and later the code), do you feel that you understand what's going on well enough to try to fix a bug if you found one?

2. What sorts of interactions between systems (or between scrub and the rest of the kernel) am I missing?

3. Do you feel confident enough in the implementation as it is now that the benefits of merging the feature (as EXPERIMENTAL) outweigh any potential disruptions to XFS at large?

4. Are there problematic interactions between subsystems that ought to be cleared up before merging?

5. Can I just merge all of this?

I intend to commit this document to the kernel's documentation directory when we start merging the patchset, albeit without the links to git.kernel.org. A much more readable version of this is posted at:
https://djwong.org/docs/xfs-online-fsck-design/

v2: add missing sections about: all the in-kernel data structures and
new apis that the scrub and repair functions use; how xattrs and directories are checked; how space btree records are checked; and add more details to the parts where all these bits tie together. Proofread for verb tense inconsistencies and eliminate vague 'we' usage. Move all the discussion of what we can do with pageable kernel memory into a single source file and section. Document where log incompat feature locks fit into the locking model.

v3: resync with 6.0, fix a few typos, begin discussion of the merging
plan for this megapatchset.

If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=online-fsck-design

---
 Documentation/filesystems/index.rst                |    1 
 .../filesystems/xfs-online-fsck-design.rst         | 4975 ++++++++++++++++++++
 .../filesystems/xfs-self-describing-metadata.rst   |    1 
 3 files changed, 4977 insertions(+)
 create mode 100644 Documentation/filesystems/xfs-online-fsck-design.rst
#2 lore raw

PATCH 1/14 xfs: document the motivation for online fsck design

From: Darrick J. Wong <[email protected]>

Start the first chapter of the online fsck design documentation. This covers the motivations for creating this in the first place.

---
 Documentation/filesystems/index.rst                |    1 
 .../filesystems/xfs-online-fsck-design.rst         |  199 ++++++++++++++++++++
 2 files changed, 200 insertions(+)
 create mode 100644 Documentation/filesystems/xfs-online-fsck-design.rst
Documentation/filesystems/index.rst +1 -0
index bee63d42e5ec..fbb2b5ada95b 100644
@@ -123,4 +123,5 @@ Documentation for filesystem implementations.
    vfat
    xfs-delayed-logging-design
    xfs-self-describing-metadata
+   xfs-online-fsck-design
    zonefs
Documentation/filesystems/xfs-online-fsck-design.rst new file +199 -0

199 lines not on this page · raw message

#3 lore raw

Re: [PATCH 01/14] xfs: document the motivation for online fsck design

Quoted 62 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the first chapter of the online fsck design documentation. This covers the motivations for creating this in the first place.

---
 Documentation/filesystems/index.rst                |    1 .../filesystems/xfs-online-fsck-design.rst         |  199
++++++++++++++++++++

 2 files changed, 200 insertions(+)

 create mode 100644 Documentation/filesystems/xfs-online-fsck- design.rst

Documentation/filesystems/index.rst +0 -0
index bee63d42e5ec..fbb2b5ada95b 100644
@@ -123,4 +123,5 @@ Documentation for filesystem implementations.

    vfat
    xfs-delayed-logging-design
    xfs-self-describing-metadata
+   xfs-online-fsck-design
    zonefs

Documentation/filesystems/xfs-online-fsck-design.rst new file +14 -0
new file mode 100644
index 000000000000..25717ebb5f80
@@ -0,0 +1,199 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. _xfs_online_fsck_design:
+
+..
+        Mapping of heading styles within this document:
+        Heading 1 uses "====" above and below
+        Heading 2 uses "===="
+        Heading 3 uses "----"
+        Heading 4 uses "````"
+        Heading 5 uses "^^^^"
+        Heading 6 uses "~~~~"
+        Heading 7 uses "...."
+
+        Sections are manually numbered because apparently that's

what everyone
+        does in the kernel.

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+======================
+XFS Online Fsck Design
+======================
+
+This document captures the design of the online filesystem check

feature for

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS.
+The purpose of this document is threefold:
+
+- To help kernel distributors understand exactly what the XFS online

fsck
+  feature is, and issues about which they should be aware.

Quoted 8 lines

+
+- To help people reading the code to familiarize themselves with the relevant
+  concepts and design points before they start digging into the code.
+
+- To help developers maintaining the system by capturing the reasons +  supporting higher level decisionmaking.

nit: decision making

Quoted 24 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+As the online fsck code is merged, the links in this document to

topic branches

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+will be replaced with links to code.
+
+This document is licensed under the terms of the GNU Public License,

v2.

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+The primary author is Darrick J. Wong.
+
+This design document is split into seven parts.
+Part 1 defines what fsck tools are and the motivations for writing a

new one.
+Parts 2 and 3 present a high level overview of how online fsck process works

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+and how it is tested to ensure correct functionality.
+Part 4 discusses the user interface and the intended usage modes of

the new

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+program.
+Parts 5 and 6 show off the high level components and how they fit

together, and
+then present case studies of how each repair function actually works.
+Part 7 sums up what has been discussed so far and speculates about what else

Documentation/filesystems/xfs-online-fsck-design.rst line 53

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +5 -0
+might be built atop online fsck.
+
+.. contents:: Table of Contents
+   :local:
+

Something that I've noticed in my training sessions is that often times, less is more. People really only absorb so much over a particular duration of time, so sometimes having too much detail in the context is not as helpful as you might think. A lot of times, paraphrasing excerpts to reflect the same info in a more compact format will help you keep audience on track (a little longer at least).

Quoted 10 lines

+1. What is a Filesystem Check?
+==============================

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+A Unix filesystem has three main jobs: to provide a hierarchy of

names through
+which application programs can associate arbitrary blobs of data for any
+length of time, to virtualize physical storage media across those names, and
+to retrieve the named data blobs at any time.

Consider the following paraphrase:

A Unix filesystem has three main jobs:
* Provide a hierarchy of names by which applications access data for a
length of time.
* Store or retrieve that data at any time.
* Virtualize physical storage media across those names

Also... I dont think it would be inappropriate to just skip the above, and jump right into fsck. That's a very limited view of a filesystem, likely a reader seeking an fsck doc probably has some idea of what a fs is otherwise supposed to be doing.

Quoted 9 lines

+The filesystem check (fsck) tool examines all the metadata in a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to look for errors.
+Simple tools only check for obvious corruptions, but the more

sophisticated

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+ones cross-reference metadata records to look for inconsistencies.
+People do not like losing data, so most fsck tools also contains

some ability
+to deal with any problems found.

While simple tools can detect data corruptions, a filesystem check (fsck) uses metadata records as a cross-reference to find and correct more inconsistencies.

?

Quoted 17 lines

+As a word of caution -- the primary goal of most Linux fsck tools is to restore
+the filesystem metadata to a consistent state, not to maximize the data

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+recovered.
+That precedent will not be challenged here.
+
+Filesystems of the 20th century generally lacked any redundancy in

the ondisk
+format, which means that fsck can only respond to errors by erasing files until

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+errors are no longer detected.
+System administrators avoid data loss by increasing the number of

separate
+storage systems through the creation of backups;

Quoted 3 lines

and they avoid downtime by
+increasing the redundancy of each storage system through the creation of RAID.

Mmm, raids help more for hardware failures right? They dont really have a notion of when the fs is corrupted. While an fsck can help navigate around a corruption possibly caused by a hardware failure, I think it's really a different kind of redundancy. I think I'd probably drop the last line and keep the selling point focused online repair.

Quoted 14 lines

+More recent filesystem designs contain enough redundancy in their metadata that
+it is now possible to regenerate data structures when non- catastrophic errors
+occur;

this capability aids both strategies.
+Over the past few years, XFS has added a storage space reverse mapping index to
+make it easy to find which files or metadata objects think they own a

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+particular range of storage.
+Efforts are under way to develop a similar reverse mapping index for
Quoted 6 lines

the naming
+hierarchy, which will involve storing directory parent pointers in each file.
+With these two pieces in place, XFS uses secondary information to perform more
+sophisticated repairs.

This part here I think I would either let go or relocate. The topic of this section is supposed to discuss roughly what a filesystem check is. Ideally so we can start talking about how ofsck is different. It feels like a bit of a jump to suddenly hop into rmap and pptrs, and for "sophisticated repairs" that we havn't really gotten into the details of yet. So I think it would read easier if we saved this part until we start talking about how they are used later.

Quoted 58 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+TLDR; Show Me the Code!
+-----------------------
+
+Code is posted to the kernel.org git trees as follows:
+`kernel changes

<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git
/log/?h=repair-symlink>`_,
+`userspace changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.
git/log/?h=scrub-media-scan-service>`_, and
+`QA test changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.
git/log/?h=repair-dirs>`_.
+Each kernel patchset adding an online repair function will use the same branch

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+name across the kernel, xfsprogs, and fstests git repos.
+
+Existing Tools
+--------------
+
+The online fsck tool described here will be the third tool in the

history of

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS (on Linux) to check and repair filesystems.
+Two programs precede it:
+
+The first program, ``xfs_check``, was created as part of the XFS

debugger

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+(``xfs_db``) and can only be used with unmounted filesystems.
+It walks all metadata in the filesystem looking for inconsistencies

in the

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+metadata, though it lacks any ability to repair what it finds.
+Due to its high memory requirements and inability to repair things,

this

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program is now deprecated and will not be discussed further.
+
+The second program, ``xfs_repair``, was created to be faster and

more robust

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+than the first program.
+Like its predecessor, it can only be used with unmounted

filesystems.
+It uses extent-based in-memory data structures to reduce memory consumption,
+and tries to schedule readahead IO appropriately to reduce I/O waiting time

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+while it scans the metadata of the entire filesystem.
+The most important feature of this tool is its ability to respond to
+inconsistencies in file metadata and directory tree by erasing

things as needed

Documentation/filesystems/xfs-online-fsck-design.rst new file +9 -0
+to eliminate problems.
+Space usage metadata are rebuilt from the observed file metadata.
+
+Problem Statement
+-----------------
+
+The current XFS tools leave several problems unsolved:
+
+1. **User programs** suddenly **lose access** to information in the
Quoted 5 lines

computer
+   when unexpected shutdowns occur as a result of silent corruptions in the
+   filesystem metadata.
+   These occur **unpredictably** and often without warning.

1. **User programs** suddenly **lose access** to the filesystem when unexpected shutdowns occur as a result of silent corruptions
that could have otherwise been avoided with an online repair

While some of these issues are not untrue, I think it makes sense to limit them to the issue you plan to solve, and therefore discuss.

Quoted 13 lines

+
+2. **Users** experience a **total loss of service** during the recovery period
+   after an **unexpected shutdown** occurs.
+
+3. **Users** experience a **total loss of service** if the filesystem is taken
+   offline to **look for problems** proactively.
+
+4. **Data owners** cannot **check the integrity** of their stored data without
+   reading all of it.

Quoted 3 lines

+   This may expose them to substantial billing costs when a linear media scan
+   might suffice.

Ok, I had to re-read this one a few times, but I think this reads a little cleaner:

Customers that are billed for data egress may incur unnecessary
cost when a background media scan on the host may have sufficed

?

Quoted 9 lines

+
+5. **System administrators** cannot **schedule** a maintenance window to deal
+   with corruptions if they **lack the means** to assess filesystem health
+   while the filesystem is online.
+
+6. **Fleet monitoring tools** cannot **automate periodic checks** of filesystem

Quoted 8 lines

+   health when doing so requires **manual intervention** and downtime.
+
+7. **Users** can be tricked into **doing things they do not desire** when
+   malicious actors **exploit quirks of Unicode** to place misleading names
+   in directories.

hrmm, I guess I'm not immediately extrapolating what things users are being tricked into doing, or how ofsck solves this? Otherwise I might drop the last one here, I think the rest of the bullets are plenty of motivation.

Quoted 34 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Given this definition of the problems to be solved and the actors

who would
+benefit, the proposed solution is a third fsck tool that acts on a running

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+filesystem.
+
+This new third program has three components: an in-kernel facility

to check
+metadata, an in-kernel facility to repair metadata, and a userspace driver

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program to drive fsck activity on a live filesystem.
+``xfs_scrub`` is the name of the driver program.
+The rest of this document presents the goals and use cases of the

new fsck
+tool, describes its major design points in connection to those goals, and

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -1
+discusses the similarities and differences with existing tools.
+
++-------------------------------------------------------------------
-------+
+|

**Note**:
    |
++-------------------------------------------------------------------
-------+
+| Throughout this document, the existing offline fsck tool can also be     |
+| referred to by its current name
"``xfs_repair``".                        |
+| The userspace driver program for the new online fsck tool can be         |
+| referred to as
"``xfs_scrub``".                                          |

Quoted 9 lines

+| The kernel portion of online fsck that validates metadata is called      |
+| "online scrub", and portion of the kernel that fixes metadata is called  |
+| "online
repair".                                                         |
++-------------------------------------------------------------------
-------+

Hmm, maybe here might be a good spot to move rmap and pptrs? It's not otherwise clear to me what "secondary metadata" is. If that is what it is meant to refer to, I think the reader will more intuitively make the connection if those two blurbs appear in the same context.

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Secondary metadata indices enable the reconstruction of parts of a

damaged
+primary metadata object from secondary information.

I would take out this blurb...

Quoted 11 lines

+XFS filesystems shard themselves into multiple primary objects to enable better
+performance on highly threaded systems and to contain the blast radius when

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+problems happen.
 
 
+The naming hierarchy is broken up into objects known as directories

and files;
+and the physical space is split into pieces known as allocation groups.

And add here:

"This enables better performance on highly threaded systems and helps to contain corruptions when they occur."

I think that reads cleaner

Quoted 18 lines

+The division of the filesystem into principal objects (allocation groups and
+inodes) means that there are ample opportunities to perform targeted checks and

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+repairs on a subset of the filesystem.
+While this is going on, other parts continue processing IO requests.
+Even if a piece of filesystem metadata can only be regenerated by

scanning the
+entire system, the scan can still be done in the background while other file

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+operations continue.
+
+In summary, online fsck takes advantage of resource sharding and

redundant
+metadata to enable targeted checking and repair operations while the system

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+is running.
+This capability will be coupled to automatic system management so
Quoted 3 lines

that
+autonomous self-healing of XFS maximizes service availability.

Nits and paraphrases aside, I think this looks pretty good?

Allison

#4 lore raw

Re: [PATCH 01/14] xfs: document the motivation for online fsck design

Quoted 64 lines · Allison Henderson
Quoted 63 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the first chapter of the online fsck design documentation. This covers the motivations for creating this in the first place.

---
 Documentation/filesystems/index.rst                |    1 .../filesystems/xfs-online-fsck-design.rst         |  199
++++++++++++++++++++

 2 files changed, 200 insertions(+)

 create mode 100644 Documentation/filesystems/xfs-online-fsck- design.rst

Documentation/filesystems/index.rst +0 -0
index bee63d42e5ec..fbb2b5ada95b 100644
@@ -123,4 +123,5 @@ Documentation for filesystem implementations.

    vfat
    xfs-delayed-logging-design
    xfs-self-describing-metadata
+   xfs-online-fsck-design
    zonefs

Documentation/filesystems/xfs-online-fsck-design.rst new file +14 -0
new file mode 100644
index 000000000000..25717ebb5f80
@@ -0,0 +1,199 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. _xfs_online_fsck_design:
+
+..
+        Mapping of heading styles within this document:
+        Heading 1 uses "====" above and below
+        Heading 2 uses "===="
+        Heading 3 uses "----"
+        Heading 4 uses "````"
+        Heading 5 uses "^^^^"
+        Heading 6 uses "~~~~"
+        Heading 7 uses "...."
+
+        Sections are manually numbered because apparently that's

what everyone
+        does in the kernel.

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+======================
+XFS Online Fsck Design
+======================
+
+This document captures the design of the online filesystem check

feature for

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS.
+The purpose of this document is threefold:
+
+- To help kernel distributors understand exactly what the XFS online

fsck
+  feature is, and issues about which they should be aware. +

Quoted 8 lines
Quoted 7 lines

+- To help people reading the code to familiarize themselves with the relevant
+  concepts and design points before they start digging into the code.
+
+- To help developers maintaining the system by capturing the reasons +  supporting higher level decisionmaking.

nit: decision making

Fixed.

Quoted 49 lines
Quoted 24 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+As the online fsck code is merged, the links in this document to

topic branches

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+will be replaced with links to code.
+
+This document is licensed under the terms of the GNU Public License,

v2.

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+The primary author is Darrick J. Wong.
+
+This design document is split into seven parts.
+Part 1 defines what fsck tools are and the motivations for writing a

new one.
+Parts 2 and 3 present a high level overview of how online fsck process works

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+and how it is tested to ensure correct functionality.
+Part 4 discusses the user interface and the intended usage modes of

the new

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+program.
+Parts 5 and 6 show off the high level components and how they fit

together, and
+then present case studies of how each repair function actually works.
+Part 7 sums up what has been discussed so far and speculates about what else

Documentation/filesystems/xfs-online-fsck-design.rst line 53

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +5 -0
+might be built atop online fsck.
+
+.. contents:: Table of Contents
+   :local:
+

Something that I've noticed in my training sessions is that often times, less is more. People really only absorb so much over a particular duration of time, so sometimes having too much detail in the context is not as helpful as you might think. A lot of times, paraphrasing excerpts to reflect the same info in a more compact format will help you keep audience on track (a little longer at least).

Quoted 10 lines

+1. What is a Filesystem Check?
+==============================

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+A Unix filesystem has three main jobs: to provide a hierarchy of

names through
+which application programs can associate arbitrary blobs of data for any
+length of time, to virtualize physical storage media across those names, and
+to retrieve the named data blobs at any time.

Consider the following paraphrase:

Quoted 5 lines

A Unix filesystem has three main jobs:
* Provide a hierarchy of names by which applications access data for a
length of time.
* Store or retrieve that data at any time.
* Virtualize physical storage media across those names

Ooh, listifying. I did quite a bit of that to break up the walls of text in earlier revisions, but apparently I missed this one.

Quoted 4 lines

Also... I dont think it would be inappropriate to just skip the above, and jump right into fsck. That's a very limited view of a filesystem, likely a reader seeking an fsck doc probably has some idea of what a fs is otherwise supposed to be doing.

This will become part of the general kernel documentation, so we can't assume that all readers are going to know what a fs really does.

"A Unix filesystem has four main responsibilities:

- Provide a hierarchy of names through which application programs can associate arbitrary blobs of data for any length of time,

- Virtualize physical storage media across those names, and

- Retrieve the named data blobs at any time.

- Examine resource usage.

"Metadata directly supporting these functions (e.g. files, directories, space mappings) are sometimes called primary metadata. Secondary metadata (e.g. reverse mapping and directory parent pointers) support operations internal to the filesystem, such as internal consistency checking and reorganization."

(I added those last two sentences in response to a point you made below.)

Quoted 14 lines
Quoted 9 lines

+The filesystem check (fsck) tool examines all the metadata in a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to look for errors.
+Simple tools only check for obvious corruptions, but the more

sophisticated

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+ones cross-reference metadata records to look for inconsistencies.
+People do not like losing data, so most fsck tools also contains

some ability
+to deal with any problems found.

While simple tools can detect data corruptions, a filesystem check (fsck) uses metadata records as a cross-reference to find and correct more inconsistencies.

Quoted 1 line

?

Let's be careful with the term 'data corruption' here -- a lot of people (well ok me) will see that as *user* data corruption, whereas we're talking about *metadata* corruption.

I think I'll rework that second sentence further:

"In addition to looking for obvious metadata corruptions, fsck also cross-references different types of metadata records with each other to look for inconsistencies."

Since the really dumb fscks of the 1970s are a long ways past now.

Quoted 17 lines
Quoted 15 lines

+As a word of caution -- the primary goal of most Linux fsck tools is to restore
+the filesystem metadata to a consistent state, not to maximize the data

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+recovered.
+That precedent will not be challenged here.
+
+Filesystems of the 20th century generally lacked any redundancy in

the ondisk
+format, which means that fsck can only respond to errors by erasing files until

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+errors are no longer detected.
+System administrators avoid data loss by increasing the number of

separate
+storage systems through the creation of backups;

Quoted 5 lines
Quoted 3 lines

and they avoid downtime by
+increasing the redundancy of each storage system through the creation of RAID.

Mmm, raids help more for hardware failures right? They dont really have a notion of when the fs is corrupted.

Right.

Quoted 4 lines

While an fsck can help
navigate around a corruption possibly caused by a hardware failure, I think it's really a different kind of redundancy. I think I'd probably drop the last line and keep the selling point focused online repair.

Yes, RAIDs provide a totally different type of redundancy. I decided to make this point specifically to counter the people who argue that RAID makes them impervious to corruption problems, etc.

This attitude seemed rather prevalent in the early days of btrfs and a certain other filesystem that Shall Not Be Named, even though the btrfs developers themselves acknowledge this distinction, given the existence of btrfs scrub' and btrfs check'.

However you do have a good point that this sentence doesn't add much where it is. I think I'll add it as a sidebar at the end of the paragraph.

Quoted 19 lines
Quoted 19 lines

+More recent filesystem designs contain enough redundancy in their metadata that
+it is now possible to regenerate data structures when non- catastrophic errors
+occur;

this capability aids both strategies.
+Over the past few years, XFS has added a storage space reverse mapping index to
+make it easy to find which files or metadata objects think they own a

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+particular range of storage.
+Efforts are under way to develop a similar reverse mapping index for

the naming
+hierarchy, which will involve storing directory parent pointers in each file.
+With these two pieces in place, XFS uses secondary information to perform more

Quoted 8 lines
Quoted 1 line

+sophisticated repairs.

This part here I think I would either let go or relocate. The topic of this section is supposed to discuss roughly what a filesystem check is. Ideally so we can start talking about how ofsck is different. It feels like a bit of a jump to suddenly hop into rmap and pptrs, and for "sophisticated repairs" that we havn't really gotten into the details of yet. So I think it would read easier if we saved this part until we start talking about how they are used later.

Agreed.

Quoted 69 lines
Quoted 58 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+TLDR; Show Me the Code!
+-----------------------
+
+Code is posted to the kernel.org git trees as follows:
+`kernel changes

<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git
/log/?h=repair-symlink>`_,
+`userspace changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.
git/log/?h=scrub-media-scan-service>`_, and
+`QA test changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.
git/log/?h=repair-dirs>`_.
+Each kernel patchset adding an online repair function will use the same branch

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+name across the kernel, xfsprogs, and fstests git repos.
+
+Existing Tools
+--------------
+
+The online fsck tool described here will be the third tool in the

history of

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS (on Linux) to check and repair filesystems.
+Two programs precede it:
+
+The first program, ``xfs_check``, was created as part of the XFS

debugger

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+(``xfs_db``) and can only be used with unmounted filesystems.
+It walks all metadata in the filesystem looking for inconsistencies

in the

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+metadata, though it lacks any ability to repair what it finds.
+Due to its high memory requirements and inability to repair things,

this

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program is now deprecated and will not be discussed further.
+
+The second program, ``xfs_repair``, was created to be faster and

more robust

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+than the first program.
+Like its predecessor, it can only be used with unmounted

filesystems.
+It uses extent-based in-memory data structures to reduce memory consumption,
+and tries to schedule readahead IO appropriately to reduce I/O waiting time

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+while it scans the metadata of the entire filesystem.
+The most important feature of this tool is its ability to respond to
+inconsistencies in file metadata and directory tree by erasing

things as needed

Documentation/filesystems/xfs-online-fsck-design.rst new file +9 -0
+to eliminate problems.
+Space usage metadata are rebuilt from the observed file metadata.
+
+Problem Statement
+-----------------
+
+The current XFS tools leave several problems unsolved:
+
+1. **User programs** suddenly **lose access** to information in the
Quoted 5 lines

computer
+   when unexpected shutdowns occur as a result of silent corruptions in the
+   filesystem metadata.
+   These occur **unpredictably** and often without warning.

1. **User programs** suddenly **lose access** to the filesystem when unexpected shutdowns occur as a result of silent corruptions
that could have otherwise been avoided with an online repair

Quoted 2 lines

While some of these issues are not untrue, I think it makes sense to limit them to the issue you plan to solve, and therefore discuss.

Fair enough, it's not like one loses /all/ the data in the computer.

That said, we're still in the problem definition phase, so I don't want to mention online repair just yet.

Quoted 15 lines
Quoted 12 lines

+2. **Users** experience a **total loss of service** during the recovery period
+   after an **unexpected shutdown** occurs.
+
+3. **Users** experience a **total loss of service** if the filesystem is taken
+   offline to **look for problems** proactively.
+
+4. **Data owners** cannot **check the integrity** of their stored data without
+   reading all of it.

Quoted 3 lines

+   This may expose them to substantial billing costs when a linear media scan
+   might suffice.

Quoted 7 lines

Ok, I had to re-read this one a few times, but I think this reads a little cleaner:

Customers that are billed for data egress may incur unnecessary
cost when a background media scan on the host may have sufficed

?

"...when a linear media scan performed by the storage system administrator would suffice."

I was tempted to say "storage owner" instead of "storage system administrator" but that sounded a little too IBM.

Quoted 12 lines
Quoted 12 lines

+5. **System administrators** cannot **schedule** a maintenance window to deal
+   with corruptions if they **lack the means** to assess filesystem health
+   while the filesystem is online.
+
+6. **Fleet monitoring tools** cannot **automate periodic checks** of filesystem
+   health when doing so requires **manual intervention** and downtime.
+
+7. **Users** can be tricked into **doing things they do not desire**

Quoted 8 lines
Quoted 4 lines

when
+   malicious actors **exploit quirks of Unicode** to place misleading names
+   in directories.

hrmm, I guess I'm not immediately extrapolating what things users are being tricked into doing, or how ofsck solves this? Otherwise I might drop the last one here, I think the rest of the bullets are plenty of motivation.

The doc gets into this later[1], but it's possible to create two entries within the same directory that have different byte sequences in the name but render identically in file choosers. These pathnames:

/home/djwong/Downloads/rustup.sh /home/djwong/Downloads/rus<zero width space>tup.sh

refer to different files, but a naïve file open dialog will render them identically as "rustup.sh". If the first is the Rust installer and the second name is actually a ransomware payload, I can victimize you by tricking you into opening the wrong one.

Firefox had a whole CVE over this in 2018:
https://bugzilla.mozilla.org/show_bug.cgi?id=1438025

xfs_scrub is (so far) the only linux filesystem fsck tool that will warn system administrators about this kind of thing.

See generic/453 and generic/454.

[1] https://djwong.org/docs/xfs-online-fsck-design/#id108

Quoted 34 lines
Quoted 34 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Given this definition of the problems to be solved and the actors

who would
+benefit, the proposed solution is a third fsck tool that acts on a running

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+filesystem.
+
+This new third program has three components: an in-kernel facility

to check
+metadata, an in-kernel facility to repair metadata, and a userspace driver

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program to drive fsck activity on a live filesystem.
+``xfs_scrub`` is the name of the driver program.
+The rest of this document presents the goals and use cases of the

new fsck
+tool, describes its major design points in connection to those goals, and

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -1
+discusses the similarities and differences with existing tools.
+
++-------------------------------------------------------------------
-------+
+|

**Note**:
    |
++-------------------------------------------------------------------
-------+
+| Throughout this document, the existing offline fsck tool can also be     |
+| referred to by its current name
"``xfs_repair``".                        |
+| The userspace driver program for the new online fsck tool can be         |
+| referred to as
"``xfs_scrub``".                                          |

Quoted 8 lines
Quoted 8 lines

+| The kernel portion of online fsck that validates metadata is called      |
+| "online scrub", and portion of the kernel that fixes metadata is called  |
+| "online
repair".                                                         |
++-------------------------------------------------------------------
-------+

Errr ^^^^ is Evolution doing line wrapping here?

Quoted 4 lines

Hmm, maybe here might be a good spot to move rmap and pptrs? It's not otherwise clear to me what "secondary metadata" is. If that is what it is meant to refer to, I think the reader will more intuitively make the connection if those two blurbs appear in the same context.

Ooh, you found a significant gap-- nowhere in this chapter do I actually define what is primary metadata. Or secondary metadata.

Quoted 19 lines
Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Secondary metadata indices enable the reconstruction of parts of a

damaged
+primary metadata object from secondary information.

I would take out this blurb...

Quoted 11 lines

+XFS filesystems shard themselves into multiple primary objects to enable better
+performance on highly threaded systems and to contain the blast radius when

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+problems happen.
 
 
+The naming hierarchy is broken up into objects known as directories

and files;
+and the physical space is split into pieces known as allocation groups.

And add here:

Quoted 4 lines

"This enables better performance on highly threaded systems and helps to contain corruptions when they occur."

I think that reads cleaner

Ok. Mind if I reword this slightly? The entire paragraph now reads like this:

"The naming hierarchy is broken up into objects known as directories and files and the physical space is split into pieces known as allocation groups. Sharding enables better performance on highly parallel systems and helps to contain the damage when corruptions occur. The division of the filesystem into principal objects (allocation groups and inodes) means that there are ample opportunities to perform targeted checks and repairs on a subset of the filesystem."

Quoted 15 lines
Quoted 15 lines

+The division of the filesystem into principal objects (allocation groups and
+inodes) means that there are ample opportunities to perform targeted checks and

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+repairs on a subset of the filesystem.
+While this is going on, other parts continue processing IO requests.
+Even if a piece of filesystem metadata can only be regenerated by

scanning the
+entire system, the scan can still be done in the background while other file

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+operations continue.
+
+In summary, online fsck takes advantage of resource sharding and

redundant
+metadata to enable targeted checking and repair operations while the

Quoted 8 lines
Quoted 6 lines

system

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+is running.
+This capability will be coupled to automatic system management so

that
+autonomous self-healing of XFS maximizes service availability.

Nits and paraphrases aside, I think this looks pretty good?

Woot. Thanks for digging in! :)

Quoted 2 lines

Allison

#5 lore raw

Re: [PATCH 01/14] xfs: document the motivation for online fsck design

Quoted 170 lines · Darrick J. Wong
Quoted 67 lines · Allison Henderson
Quoted 66 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the first chapter of the online fsck design documentation. This covers the motivations for creating this in the first place.

---
 Documentation/filesystems/index.rst                |    1 .../filesystems/xfs-online-fsck-design.rst         |  199
++++++++++++++++++++

 2 files changed, 200 insertions(+)

 create mode 100644 Documentation/filesystems/xfs-online-fsck- design.rst

Documentation/filesystems/index.rst +0 -0
index bee63d42e5ec..fbb2b5ada95b 100644
@@ -123,4 +123,5 @@ Documentation for filesystem implementations.

    vfat
    xfs-delayed-logging-design
    xfs-self-describing-metadata
+   xfs-online-fsck-design
    zonefs

Documentation/filesystems/xfs-online-fsck-design.rst new file +14 -0
new file mode 100644
index 000000000000..25717ebb5f80
@@ -0,0 +1,199 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. _xfs_online_fsck_design:
+
+..
+        Mapping of heading styles within this document:
+        Heading 1 uses "====" above and below
+        Heading 2 uses "===="
+        Heading 3 uses "----"
+        Heading 4 uses "````"
+        Heading 5 uses "^^^^"
+        Heading 6 uses "~~~~"
+        Heading 7 uses "...."
+
+        Sections are manually numbered because apparently that's

what everyone
+        does in the kernel.

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+======================
+XFS Online Fsck Design
+======================
+
+This document captures the design of the online filesystem check

feature for

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS.
+The purpose of this document is threefold:
+
+- To help kernel distributors understand exactly what the XFS

online
fsck
+  feature is, and issues about which they should be aware. +
+- To help people reading the code to familiarize themselves with the

Quoted 8 lines
Quoted 7 lines

relevant
+  concepts and design points before they start digging into the code.
+
+- To help developers maintaining the system by capturing the reasons
+  supporting higher level decisionmaking.

nit: decision making

Fixed.

Quoted 58 lines
Quoted 29 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+As the online fsck code is merged, the links in this document to

topic branches

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+will be replaced with links to code.
+
+This document is licensed under the terms of the GNU Public

License,
v2.

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+The primary author is Darrick J. Wong.
+
+This design document is split into seven parts.
+Part 1 defines what fsck tools are and the motivations for

writing a
new one.
+Parts 2 and 3 present a high level overview of how online fsck process works

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+and how it is tested to ensure correct functionality.
+Part 4 discusses the user interface and the intended usage modes

of
the new

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+program.
+Parts 5 and 6 show off the high level components and how they

fit
together, and
+then present case studies of how each repair function actually works.
+Part 7 sums up what has been discussed so far and speculates about
what else

Documentation/filesystems/xfs-online-fsck-design.rst line 53

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +5 -0
+might be built atop online fsck.
+
+.. contents:: Table of Contents
+   :local:
+

Something that I've noticed in my training sessions is that often times, less is more.  People really only absorb so much over a particular duration of time, so sometimes having too much detail in the
context is not as helpful as you might think.  A lot of times, paraphrasing excerpts to reflect the same info in a more compact format
will help you keep audience on track (a little longer at least).

Quoted 4 lines

+1. What is a Filesystem Check?
+==============================

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+A Unix filesystem has three main jobs: to provide a hierarchy of
Quoted 8 lines

names through
+which application programs can associate arbitrary blobs of data for
any
+length of time, to virtualize physical storage media across those
names, and
+to retrieve the named data blobs at any time.

Consider the following paraphrase:

Quoted 6 lines

A Unix filesystem has three main jobs:
 * Provide a hierarchy of names by which applications access data for a
length of time.
 * Store or retrieve that data at any time.
 * Virtualize physical storage media across those names

Ooh, listifying.  I did quite a bit of that to break up the walls of text in earlier revisions, but apparently I missed this one.

Quoted 7 lines

Also... I dont think it would be inappropriate to just skip the above,
and jump right into fsck.  That's a very limited view of a filesystem,
likely a reader seeking an fsck doc probably has some idea of what a fs
is otherwise supposed to be doing.

This will become part of the general kernel documentation, so we can't
assume that all readers are going to know what a fs really does.

"A Unix filesystem has four main responsibilities:

- Provide a hierarchy of names through which application programs can associate arbitrary blobs of data for any length of time,

- Virtualize physical storage media across those names, and

- Retrieve the named data blobs at any time.

- Examine resource usage.

Quoted 7 lines

"Metadata directly supporting these functions (e.g. files, directories,
space mappings) are sometimes called primary metadata. Secondary metadata (e.g. reverse mapping and directory parent pointers)
support operations internal to the filesystem, such as internal consistency checking and reorganization."

Sure, I think that sounds good and helps to set up the metadata concepts that are discussed later.

Quoted 27 lines

(I added those last two sentences in response to a point you made below.)

Quoted 16 lines
Quoted 10 lines

+The filesystem check (fsck) tool examines all the metadata in a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to look for errors.
+Simple tools only check for obvious corruptions, but the more

sophisticated
+ones cross-reference metadata records to look for inconsistencies.
+People do not like losing data, so most fsck tools also contains some ability
+to deal with any problems found.

While simple tools can detect data corruptions, a filesystem check (fsck) uses metadata records as a cross-reference to find and correct
more inconsistencies.

Quoted 1 line

?

Let's be careful with the term 'data corruption' here -- a lot of people
(well ok me) will see that as *user* data corruption, whereas we're talking about *metadata* corruption.

Quoted 7 lines

I think I'll rework that second sentence further:

"In addition to looking for obvious metadata corruptions, fsck also cross-references different types of metadata records with each other to
look for inconsistencies."

Alrighty, that sounds good

Quoted 210 lines

Since the really dumb fscks of the 1970s are a long ways past now.

Quoted 22 lines
Quoted 20 lines

+As a word of caution -- the primary goal of most Linux fsck tools is
to restore
+the filesystem metadata to a consistent state, not to maximize the
data

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+recovered.
+That precedent will not be challenged here.
+
+Filesystems of the 20th century generally lacked any redundancy

in
the ondisk
+format, which means that fsck can only respond to errors by erasing
files until

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+errors are no longer detected.
+System administrators avoid data loss by increasing the number

of
separate
+storage systems through the creation of backups;

Quoted 5 lines
Quoted 3 lines

and they avoid downtime by
+increasing the redundancy of each storage system through the creation of RAID.

Mmm, raids help more for hardware failures right?  They dont really have a notion of when the fs is corrupted.

Right.

Quoted 7 lines

While an fsck can help
navigate around a corruption possibly caused by a hardware failure, I
think it's really a different kind of redundancy. I think I'd probably
drop the last line and keep the selling point focused online repair.

Yes, RAIDs provide a totally different type of redundancy.  I decided to
make this point specifically to counter the people who argue that RAID
makes them impervious to corruption problems, etc.

This attitude seemed rather prevalent in the early days of btrfs and a
certain other filesystem that Shall Not Be Named, even though the btrfs
developers themselves acknowledge this distinction, given the existence
of btrfs scrub' and btrfs check'.

However you do have a good point that this sentence doesn't add much where it is.  I think I'll add it as a sidebar at the end of the paragraph.

Quoted 29 lines
Quoted 17 lines

+More recent filesystem designs contain enough redundancy in their
metadata that
+it is now possible to regenerate data structures when non- catastrophic errors
+occur;

this capability aids both strategies.
+Over the past few years, XFS has added a storage space reverse mapping index to
+make it easy to find which files or metadata objects think they own
a

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+particular range of storage.
+Efforts are under way to develop a similar reverse mapping index

for

Quoted 8 lines

the naming
+hierarchy, which will involve storing directory parent pointers in
each file.
+With these two pieces in place, XFS uses secondary information to
perform more
+sophisticated repairs.

This part here I think I would either let go or relocate.  The topic of
this section is supposed to discuss roughly what a filesystem check is.

Quoted 8 lines

Ideally so we can start talking about how ofsck is different.  It feels
like a bit of a jump to suddenly hop into rmap and pptrs, and for "sophisticated repairs" that we havn't really gotten into the details
of yet.  So I think it would read easier if we saved this part until we
start talking about how they are used later.

Agreed.

Quoted 82 lines
Quoted 68 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+TLDR; Show Me the Code!
+-----------------------
+
+Code is posted to the kernel.org git trees as follows:
+`kernel changes

<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.g
it
/log/?h=repair-symlink>`_,
+`userspace changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-
dev.
git/log/?h=scrub-media-scan-service>`_, and
+`QA test changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-
dev.
git/log/?h=repair-dirs>`_.
+Each kernel patchset adding an online repair function will use the
same branch

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+name across the kernel, xfsprogs, and fstests git repos.
+
+Existing Tools
+--------------
+
+The online fsck tool described here will be the third tool in

the
history of

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS (on Linux) to check and repair filesystems.
+Two programs precede it:
+
+The first program, ``xfs_check``, was created as part of the XFS

debugger

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+(``xfs_db``) and can only be used with unmounted filesystems.
+It walks all metadata in the filesystem looking for

inconsistencies
in the

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+metadata, though it lacks any ability to repair what it finds.
+Due to its high memory requirements and inability to repair

things,
this

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program is now deprecated and will not be discussed further.
+
+The second program, ``xfs_repair``, was created to be faster and

more robust

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+than the first program.
+Like its predecessor, it can only be used with unmounted

filesystems.
+It uses extent-based in-memory data structures to reduce memory consumption,
+and tries to schedule readahead IO appropriately to reduce I/O waiting time

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+while it scans the metadata of the entire filesystem.
+The most important feature of this tool is its ability to

respond to
+inconsistencies in file metadata and directory tree by erasing things as needed

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to eliminate problems.
+Space usage metadata are rebuilt from the observed file

metadata.

Documentation/filesystems/xfs-online-fsck-design.rst new file +7 -0
+
+Problem Statement
+-----------------
+
+The current XFS tools leave several problems unsolved:
+
+1. **User programs** suddenly **lose access** to information in
Quoted 7 lines

the
computer
+   when unexpected shutdowns occur as a result of silent corruptions
in the
+   filesystem metadata.
+   These occur **unpredictably** and often without warning.

1. **User programs** suddenly **lose access** to the filesystem when unexpected shutdowns occur as a result of silent corruptions
that could have otherwise been avoided with an online repair

Quoted 3 lines

While some of these issues are not untrue, I think it makes sense to
limit them to the issue you plan to solve, and therefore discuss.

Fair enough, it's not like one loses /all/ the data in the computer.

That said, we're still in the problem definition phase, so I don't want
to mention online repair just yet.

Quoted 20 lines
Quoted 13 lines

+2. **Users** experience a **total loss of service** during the recovery period
+   after an **unexpected shutdown** occurs.
+
+3. **Users** experience a **total loss of service** if the filesystem is taken
+   offline to **look for problems** proactively.
+
+4. **Data owners** cannot **check the integrity** of their stored
data without
+   reading all of it.

Quoted 4 lines

+   This may expose them to substantial billing costs when a linear
media scan
+   might suffice.

Ok, I had to re-read this one a few times, but I think this reads a little cleaner:

Quoted 8 lines
Quoted 4 lines

    Customers that are billed for data egress may incur unnecessary cost when a background media scan on the host may have sufficed

?

"...when a linear media scan performed by the storage system administrator would suffice."

That sounds fine to me

Quoted 49 lines

I was tempted to say "storage owner" instead of "storage system administrator" but that sounded a little too IBM.

Quoted 18 lines
Quoted 18 lines

+5. **System administrators** cannot **schedule** a maintenance window to deal
+   with corruptions if they **lack the means** to assess filesystem
health
+   while the filesystem is online.
+
+6. **Fleet monitoring tools** cannot **automate periodic checks** of
filesystem
+   health when doing so requires **manual intervention** and downtime.
+
+7. **Users** can be tricked into **doing things they do not desire**
when
+   malicious actors **exploit quirks of Unicode** to place misleading names

Quoted 8 lines
Quoted 1 line

+   in directories.

hrmm, I guess I'm not immediately extrapolating what things users are
being tricked into doing, or how ofsck solves this?  Otherwise I might
drop the last one here, I think the rest of the bullets are plenty of
motivation.

The doc gets into this later[1], but it's possible to create two entries
within the same directory that have different byte sequences in the name
but render identically in file choosers.  These pathnames:

/home/djwong/Downloads/rustup.sh /home/djwong/Downloads/rus<zero width space>tup.sh

refer to different files, but a naïve file open dialog will render them
identically as "rustup.sh".  If the first is the Rust installer and the
second name is actually a ransomware payload, I can victimize you by tricking you into opening the wrong one.

Firefox had a whole CVE over this in 2018:
https://bugzilla.mozilla.org/show_bug.cgi?id=1438025

Quoted 8 lines

xfs_scrub is (so far) the only linux filesystem fsck tool that will warn
system administrators about this kind of thing.

See generic/453 and generic/454.

[1] https://djwong.org/docs/xfs-online-fsck-design/#id108

hmm ok, how about:

7. Malicious attacks may use uncommon unicode characters to create file names that resemble normal files, which may go undetected until the filesystem is scanned.

?

Quoted 100 lines
Quoted 46 lines
Quoted 46 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Given this definition of the problems to be solved and the

actors
who would
+benefit, the proposed solution is a third fsck tool that acts on a
running

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+filesystem.
+
+This new third program has three components: an in-kernel

facility
to check
+metadata, an in-kernel facility to repair metadata, and a userspace
driver

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program to drive fsck activity on a live filesystem.
+``xfs_scrub`` is the name of the driver program.
+The rest of this document presents the goals and use cases of

the
new fsck
+tool, describes its major design points in connection to those goals, and

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -2
+discusses the similarities and differences with existing tools.
+
++---------------------------------------------------------------
----
-------+
+|

**Note**:

    |
++---------------------------------------------------------------
----
-------+
+| Throughout this document, the existing offline fsck tool can also
be     |
+| referred to by its current name
"``xfs_repair``".                        |
+| The userspace driver program for the new online fsck tool can be         |
+| referred to as
"``xfs_scrub``".                                          | +| The kernel portion of online fsck that validates metadata is called      |
+| "online scrub", and portion of the kernel that fixes metadata

Quoted 8 lines
Quoted 8 lines

is
called  |
+| "online
repair".
|
++---------------------------------------------------------------
----
-------+

Errr ^^^^ is Evolution doing line wrapping here?

Quoted 7 lines

Hmm, maybe here might be a good spot to move rmap and pptrs?  It's not
otherwise clear to me what "secondary metadata" is.  If that is what it
is meant to refer to, I think the reader will more intuitively make the
connection if those two blurbs appear in the same context.

Ooh, you found a significant gap-- nowhere in this chapter do I actually
define what is primary metadata.  Or secondary metadata.

Quoted 22 lines
Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Secondary metadata indices enable the reconstruction of parts of

a
damaged
+primary metadata object from secondary information.

I would take out this blurb...

Quoted 9 lines

+XFS filesystems shard themselves into multiple primary objects to
enable better
+performance on highly threaded systems and to contain the blast radius when

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+problems happen.
 
 
+The naming hierarchy is broken up into objects known as
Quoted 4 lines

directories
and files;
+and the physical space is split into pieces known as allocation groups.

And add here:

Quoted 5 lines

"This enables better performance on highly threaded systems and helps
to contain corruptions when they occur."

I think that reads cleaner

Ok.  Mind if I reword this slightly?  The entire paragraph now reads like this:

Quoted 11 lines

"The naming hierarchy is broken up into objects known as directories and
files and the physical space is split into pieces known as allocation groups.  Sharding enables better performance on highly parallel systems
and helps to contain the damage when corruptions occur.  The division of
the filesystem into principal objects (allocation groups and inodes) means that there are ample opportunities to perform targeted checks and
repairs on a subset of the filesystem."

I think that sounds cleaner

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines

+The division of the filesystem into principal objects (allocation
groups and
+inodes) means that there are ample opportunities to perform targeted
checks and

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+repairs on a subset of the filesystem.
+While this is going on, other parts continue processing IO

requests.
+Even if a piece of filesystem metadata can only be regenerated by
scanning the
+entire system, the scan can still be done in the background while
other file

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+operations continue.
+
+In summary, online fsck takes advantage of resource sharding and

redundant
+metadata to enable targeted checking and repair operations while the
system

Quoted 8 lines
Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+is running.
+This capability will be coupled to automatic system management

so
that
+autonomous self-healing of XFS maximizes service availability.

Nits and paraphrases aside, I think this looks pretty good?

Quoted 2 lines

Woot.  Thanks for digging in! :)

Sure, no problem!

Quoted 2 lines
Quoted 2 lines

Allison

#6 lore raw

Re: [PATCH 01/14] xfs: document the motivation for online fsck design

Quoted 491 lines · Allison Henderson
Quoted 170 lines · Darrick J. Wong
Quoted 67 lines · Allison Henderson
Quoted 66 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the first chapter of the online fsck design documentation. This covers the motivations for creating this in the first place.

---
 Documentation/filesystems/index.rst                |    1 .../filesystems/xfs-online-fsck-design.rst         |  199
++++++++++++++++++++

 2 files changed, 200 insertions(+)

 create mode 100644 Documentation/filesystems/xfs-online-fsck- design.rst

Documentation/filesystems/index.rst +0 -0
index bee63d42e5ec..fbb2b5ada95b 100644
@@ -123,4 +123,5 @@ Documentation for filesystem implementations.

    vfat
    xfs-delayed-logging-design
    xfs-self-describing-metadata
+   xfs-online-fsck-design
    zonefs

Documentation/filesystems/xfs-online-fsck-design.rst new file +14 -0
new file mode 100644
index 000000000000..25717ebb5f80
@@ -0,0 +1,199 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. _xfs_online_fsck_design:
+
+..
+        Mapping of heading styles within this document:
+        Heading 1 uses "====" above and below
+        Heading 2 uses "===="
+        Heading 3 uses "----"
+        Heading 4 uses "````"
+        Heading 5 uses "^^^^"
+        Heading 6 uses "~~~~"
+        Heading 7 uses "...."
+
+        Sections are manually numbered because apparently that's

what everyone
+        does in the kernel.

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+======================
+XFS Online Fsck Design
+======================
+
+This document captures the design of the online filesystem check

feature for

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS.
+The purpose of this document is threefold:
+
+- To help kernel distributors understand exactly what the XFS

online
fsck
+  feature is, and issues about which they should be aware. +
+- To help people reading the code to familiarize themselves with the

Quoted 8 lines
Quoted 7 lines

relevant
+  concepts and design points before they start digging into the code.
+
+- To help developers maintaining the system by capturing the reasons
+  supporting higher level decisionmaking.

nit: decision making

Fixed.

Quoted 58 lines
Quoted 29 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+As the online fsck code is merged, the links in this document to

topic branches

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+will be replaced with links to code.
+
+This document is licensed under the terms of the GNU Public

License,
v2.

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+The primary author is Darrick J. Wong.
+
+This design document is split into seven parts.
+Part 1 defines what fsck tools are and the motivations for

writing a
new one.
+Parts 2 and 3 present a high level overview of how online fsck process works

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+and how it is tested to ensure correct functionality.
+Part 4 discusses the user interface and the intended usage modes

of
the new

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+program.
+Parts 5 and 6 show off the high level components and how they

fit
together, and
+then present case studies of how each repair function actually works.
+Part 7 sums up what has been discussed so far and speculates about
what else

Documentation/filesystems/xfs-online-fsck-design.rst line 53

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +5 -0
+might be built atop online fsck.
+
+.. contents:: Table of Contents
+   :local:
+

Something that I've noticed in my training sessions is that often times, less is more.  People really only absorb so much over a particular duration of time, so sometimes having too much detail in the
context is not as helpful as you might think.  A lot of times, paraphrasing excerpts to reflect the same info in a more compact format
will help you keep audience on track (a little longer at least).

Quoted 4 lines

+1. What is a Filesystem Check?
+==============================

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+A Unix filesystem has three main jobs: to provide a hierarchy of
Quoted 8 lines

names through
+which application programs can associate arbitrary blobs of data for
any
+length of time, to virtualize physical storage media across those
names, and
+to retrieve the named data blobs at any time.

Consider the following paraphrase:

Quoted 6 lines

A Unix filesystem has three main jobs:
 * Provide a hierarchy of names by which applications access data for a
length of time.
 * Store or retrieve that data at any time.
 * Virtualize physical storage media across those names

Ooh, listifying.  I did quite a bit of that to break up the walls of text in earlier revisions, but apparently I missed this one.

Quoted 7 lines

Also... I dont think it would be inappropriate to just skip the above,
and jump right into fsck.  That's a very limited view of a filesystem,
likely a reader seeking an fsck doc probably has some idea of what a fs
is otherwise supposed to be doing.

This will become part of the general kernel documentation, so we can't
assume that all readers are going to know what a fs really does.

"A Unix filesystem has four main responsibilities:

- Provide a hierarchy of names through which application programs can associate arbitrary blobs of data for any length of time,

- Virtualize physical storage media across those names, and

- Retrieve the named data blobs at any time.

- Examine resource usage.

Quoted 7 lines

"Metadata directly supporting these functions (e.g. files, directories,
space mappings) are sometimes called primary metadata. Secondary metadata (e.g. reverse mapping and directory parent pointers)
support operations internal to the filesystem, such as internal consistency checking and reorganization."

Sure, I think that sounds good and helps to set up the metadata concepts that are discussed later.

Quoted 27 lines

(I added those last two sentences in response to a point you made below.)

Quoted 16 lines
Quoted 10 lines

+The filesystem check (fsck) tool examines all the metadata in a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to look for errors.
+Simple tools only check for obvious corruptions, but the more

sophisticated
+ones cross-reference metadata records to look for inconsistencies.
+People do not like losing data, so most fsck tools also contains some ability
+to deal with any problems found.

While simple tools can detect data corruptions, a filesystem check (fsck) uses metadata records as a cross-reference to find and correct
more inconsistencies.

Quoted 1 line

?

Let's be careful with the term 'data corruption' here -- a lot of people
(well ok me) will see that as *user* data corruption, whereas we're talking about *metadata* corruption.

Quoted 7 lines

I think I'll rework that second sentence further:

"In addition to looking for obvious metadata corruptions, fsck also cross-references different types of metadata records with each other to
look for inconsistencies."

Alrighty, that sounds good

Quoted 210 lines

Since the really dumb fscks of the 1970s are a long ways past now.

Quoted 22 lines
Quoted 20 lines

+As a word of caution -- the primary goal of most Linux fsck tools is
to restore
+the filesystem metadata to a consistent state, not to maximize the
data

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+recovered.
+That precedent will not be challenged here.
+
+Filesystems of the 20th century generally lacked any redundancy

in
the ondisk
+format, which means that fsck can only respond to errors by erasing
files until

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+errors are no longer detected.
+System administrators avoid data loss by increasing the number

of
separate
+storage systems through the creation of backups;

Quoted 5 lines
Quoted 3 lines

and they avoid downtime by
+increasing the redundancy of each storage system through the creation of RAID.

Mmm, raids help more for hardware failures right?  They dont really have a notion of when the fs is corrupted.

Right.

Quoted 7 lines

While an fsck can help
navigate around a corruption possibly caused by a hardware failure, I
think it's really a different kind of redundancy. I think I'd probably
drop the last line and keep the selling point focused online repair.

Yes, RAIDs provide a totally different type of redundancy.  I decided to
make this point specifically to counter the people who argue that RAID
makes them impervious to corruption problems, etc.

This attitude seemed rather prevalent in the early days of btrfs and a
certain other filesystem that Shall Not Be Named, even though the btrfs
developers themselves acknowledge this distinction, given the existence
of btrfs scrub' and btrfs check'.

However you do have a good point that this sentence doesn't add much where it is.  I think I'll add it as a sidebar at the end of the paragraph.

Quoted 29 lines
Quoted 17 lines

+More recent filesystem designs contain enough redundancy in their
metadata that
+it is now possible to regenerate data structures when non- catastrophic errors
+occur;

this capability aids both strategies.
+Over the past few years, XFS has added a storage space reverse mapping index to
+make it easy to find which files or metadata objects think they own
a

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+particular range of storage.
+Efforts are under way to develop a similar reverse mapping index

for

Quoted 8 lines

the naming
+hierarchy, which will involve storing directory parent pointers in
each file.
+With these two pieces in place, XFS uses secondary information to
perform more
+sophisticated repairs.

This part here I think I would either let go or relocate.  The topic of
this section is supposed to discuss roughly what a filesystem check is.

Quoted 8 lines

Ideally so we can start talking about how ofsck is different.  It feels
like a bit of a jump to suddenly hop into rmap and pptrs, and for "sophisticated repairs" that we havn't really gotten into the details
of yet.  So I think it would read easier if we saved this part until we
start talking about how they are used later.

Agreed.

Quoted 82 lines
Quoted 68 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+
+TLDR; Show Me the Code!
+-----------------------
+
+Code is posted to the kernel.org git trees as follows:
+`kernel changes

<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.g
it
/log/?h=repair-symlink>`_,
+`userspace changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-
dev.
git/log/?h=scrub-media-scan-service>`_, and
+`QA test changes
<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-
dev.
git/log/?h=repair-dirs>`_.
+Each kernel patchset adding an online repair function will use the
same branch

Documentation/filesystems/xfs-online-fsck-design.rst new file +6 -0
+name across the kernel, xfsprogs, and fstests git repos.
+
+Existing Tools
+--------------
+
+The online fsck tool described here will be the third tool in

the
history of

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -0
+XFS (on Linux) to check and repair filesystems.
+Two programs precede it:
+
+The first program, ``xfs_check``, was created as part of the XFS

debugger

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+(``xfs_db``) and can only be used with unmounted filesystems.
+It walks all metadata in the filesystem looking for

inconsistencies
in the

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+metadata, though it lacks any ability to repair what it finds.
+Due to its high memory requirements and inability to repair

things,
this

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program is now deprecated and will not be discussed further.
+
+The second program, ``xfs_repair``, was created to be faster and

more robust

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+than the first program.
+Like its predecessor, it can only be used with unmounted

filesystems.
+It uses extent-based in-memory data structures to reduce memory consumption,
+and tries to schedule readahead IO appropriately to reduce I/O waiting time

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+while it scans the metadata of the entire filesystem.
+The most important feature of this tool is its ability to

respond to
+inconsistencies in file metadata and directory tree by erasing things as needed

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+to eliminate problems.
+Space usage metadata are rebuilt from the observed file

metadata.

Documentation/filesystems/xfs-online-fsck-design.rst new file +7 -0
+
+Problem Statement
+-----------------
+
+The current XFS tools leave several problems unsolved:
+
+1. **User programs** suddenly **lose access** to information in
Quoted 7 lines

the
computer
+   when unexpected shutdowns occur as a result of silent corruptions
in the
+   filesystem metadata.
+   These occur **unpredictably** and often without warning.

1. **User programs** suddenly **lose access** to the filesystem when unexpected shutdowns occur as a result of silent corruptions
that could have otherwise been avoided with an online repair

Quoted 3 lines

While some of these issues are not untrue, I think it makes sense to
limit them to the issue you plan to solve, and therefore discuss.

Fair enough, it's not like one loses /all/ the data in the computer.

That said, we're still in the problem definition phase, so I don't want
to mention online repair just yet.

Quoted 20 lines
Quoted 13 lines

+2. **Users** experience a **total loss of service** during the recovery period
+   after an **unexpected shutdown** occurs.
+
+3. **Users** experience a **total loss of service** if the filesystem is taken
+   offline to **look for problems** proactively.
+
+4. **Data owners** cannot **check the integrity** of their stored
data without
+   reading all of it.

Quoted 4 lines

+   This may expose them to substantial billing costs when a linear
media scan
+   might suffice.

Ok, I had to re-read this one a few times, but I think this reads a little cleaner:

Quoted 8 lines
Quoted 4 lines

    Customers that are billed for data egress may incur unnecessary cost when a background media scan on the host may have sufficed

?

"...when a linear media scan performed by the storage system administrator would suffice."

That sounds fine to me

Quoted 55 lines

I was tempted to say "storage owner" instead of "storage system administrator" but that sounded a little too IBM.

Quoted 18 lines
Quoted 18 lines

+5. **System administrators** cannot **schedule** a maintenance window to deal
+   with corruptions if they **lack the means** to assess filesystem
health
+   while the filesystem is online.
+
+6. **Fleet monitoring tools** cannot **automate periodic checks** of
filesystem
+   health when doing so requires **manual intervention** and downtime.
+
+7. **Users** can be tricked into **doing things they do not desire**
when
+   malicious actors **exploit quirks of Unicode** to place misleading names

Quoted 8 lines
Quoted 1 line

+   in directories.

hrmm, I guess I'm not immediately extrapolating what things users are
being tricked into doing, or how ofsck solves this?  Otherwise I might
drop the last one here, I think the rest of the bullets are plenty of
motivation.

The doc gets into this later[1], but it's possible to create two entries
within the same directory that have different byte sequences in the name
but render identically in file choosers.  These pathnames:

/home/djwong/Downloads/rustup.sh /home/djwong/Downloads/rus<zero width space>tup.sh

refer to different files, but a naïve file open dialog will render them
identically as "rustup.sh".  If the first is the Rust installer and the
second name is actually a ransomware payload, I can victimize you by tricking you into opening the wrong one.

Firefox had a whole CVE over this in 2018:
https://bugzilla.mozilla.org/show_bug.cgi?id=1438025

xfs_scrub is (so far) the only linux filesystem fsck tool that will warn
system administrators about this kind of thing.

See generic/453 and generic/454.

Quoted 7 lines
Quoted 2 lines

[1] https://djwong.org/docs/xfs-online-fsck-design/#id108

hmm ok, how about:

7. Malicious attacks may use uncommon unicode characters to create file names that resemble normal files, which may go undetected until the filesystem is scanned.

They resemble *other filenames* in the same directory, normal or otherwise.

Note that xattrs have the same problem -- a listing of attrs will show two names that render identically but map to different things. There's less double-click danger there, at least.

Another class of unicode problem is that you can use directional controls to spoof file extensions. The sequence:

pu<right to left>txt.pl

renders as "pulp.txt" if you're not careful, but file managers think it's actually a perl script file! Granted, nobody should allow execution of random a-x downloaded scripts.

There are enough weird twists to this sort of deception that I left #7 worded as broadly as I needed.

--D

Quoted 155 lines

?

Quoted 100 lines
Quoted 46 lines
Quoted 46 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Given this definition of the problems to be solved and the

actors
who would
+benefit, the proposed solution is a third fsck tool that acts on a
running

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+filesystem.
+
+This new third program has three components: an in-kernel

facility
to check
+metadata, an in-kernel facility to repair metadata, and a userspace
driver

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+program to drive fsck activity on a live filesystem.
+``xfs_scrub`` is the name of the driver program.
+The rest of this document presents the goals and use cases of

the
new fsck
+tool, describes its major design points in connection to those goals, and

Documentation/filesystems/xfs-online-fsck-design.rst new file +4 -2
+discusses the similarities and differences with existing tools.
+
++---------------------------------------------------------------
----
-------+
+|

**Note**:

    |
++---------------------------------------------------------------
----
-------+
+| Throughout this document, the existing offline fsck tool can also
be     |
+| referred to by its current name
"``xfs_repair``".                        |
+| The userspace driver program for the new online fsck tool can be         |
+| referred to as
"``xfs_scrub``".                                          | +| The kernel portion of online fsck that validates metadata is called      |
+| "online scrub", and portion of the kernel that fixes metadata

Quoted 8 lines
Quoted 8 lines

is
called  |
+| "online
repair".
|
++---------------------------------------------------------------
----
-------+

Errr ^^^^ is Evolution doing line wrapping here?

Quoted 7 lines

Hmm, maybe here might be a good spot to move rmap and pptrs?  It's not
otherwise clear to me what "secondary metadata" is.  If that is what it
is meant to refer to, I think the reader will more intuitively make the
connection if those two blurbs appear in the same context.

Ooh, you found a significant gap-- nowhere in this chapter do I actually
define what is primary metadata.  Or secondary metadata.

Quoted 22 lines
Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+
+Secondary metadata indices enable the reconstruction of parts of

a
damaged
+primary metadata object from secondary information.

I would take out this blurb...

Quoted 9 lines

+XFS filesystems shard themselves into multiple primary objects to
enable better
+performance on highly threaded systems and to contain the blast radius when

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+problems happen.
 
 
+The naming hierarchy is broken up into objects known as
Quoted 4 lines

directories
and files;
+and the physical space is split into pieces known as allocation groups.

And add here:

Quoted 5 lines

"This enables better performance on highly threaded systems and helps
to contain corruptions when they occur."

I think that reads cleaner

Ok.  Mind if I reword this slightly?  The entire paragraph now reads like this:

Quoted 11 lines

"The naming hierarchy is broken up into objects known as directories and
files and the physical space is split into pieces known as allocation groups.  Sharding enables better performance on highly parallel systems
and helps to contain the damage when corruptions occur.  The division of
the filesystem into principal objects (allocation groups and inodes) means that there are ample opportunities to perform targeted checks and
repairs on a subset of the filesystem."

I think that sounds cleaner

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines

+The division of the filesystem into principal objects (allocation
groups and
+inodes) means that there are ample opportunities to perform targeted
checks and

Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+repairs on a subset of the filesystem.
+While this is going on, other parts continue processing IO

requests.
+Even if a piece of filesystem metadata can only be regenerated by
scanning the
+entire system, the scan can still be done in the background while
other file

Documentation/filesystems/xfs-online-fsck-design.rst new file +3 -0
+operations continue.
+
+In summary, online fsck takes advantage of resource sharding and

redundant
+metadata to enable targeted checking and repair operations while the
system

Quoted 8 lines
Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst new file +2 -0
+is running.
+This capability will be coupled to automatic system management

so
that
+autonomous self-healing of XFS maximizes service availability.

Nits and paraphrases aside, I think this looks pretty good?

Quoted 2 lines

Woot.  Thanks for digging in! :)

Sure, no problem!

Quoted 2 lines
Quoted 2 lines

Allison

#7 lore raw

Re: [PATCH 01/14] xfs: document the motivation for online fsck design

In reply to Allison Henderson

On Sat, Jan 07, 2023 at 05:01:54AM +0000, Allison Henderson wrote:

<snip> There was one part of your reply that I wanted to handle separately:

Quoted 4 lines

Something that I've noticed in my training sessions is that often times, less is more. People really only absorb so much over a particular duration of time, so sometimes having too much detail in the context is not as helpful as you might think.

I'm very worried about this ^^^ exact problem making it more difficult to merge online fsck.

As the online fsck patchset grew and grew and grew, I decided that it was absolutely necessary to write a design document to condense the information from 1200 patches, for this is the diffstat for the code changes themselves:

225 files changed, 41244 insertions(+), 4388 deletions(-)
205 files changed, 16802 insertions(+), 3405 deletions(-)
438 files changed, 20123 insertions(+), 446 deletions(-)

That's 78169 insertions and 8239 deletions, or about ~70k new LOC, and that doesn't include the scrub code that's already upstream (~60000). It's wild that online fsck is larger than the filesystem.

You might recall that I sent it out for review twice last year, and the feedback I got from the experienced folk was that I needed to write in much more detail about the design -- specifically, what I was doing with the fs hooks, and all the data structures that I was layering atop tmpfs files to support rebuilds.

Before I even got to /that/ point, the design documentation had reached 4500 lines (or 90 pages) long, at which point I decided that it was necessary to write a summary to condense the 4500 lines down to a single chapter.

Hence part 1 about what is a filesystem check. It's supposed to introduce the very very broad concepts to a reader before they dive into successively higher levels of detail in the later parts.

My guess is that the audience for the code deluges and this design doc fall into roughly these categories:

* Experienced people who have been around XFS and Linux for a very long time. These people, I think, would benefit from scanning parts 2 and 3 as a refresher. Then they can scan parts 5 and 6 before moving on to the code.

* Intermediate people, who probably need to read parts 2 - 6 and understand them thoroughly before reading the code. The case studies in part 5 should be used as a guide to the patchsets.

* People who have no idea what filesystems and fsck are, want to know about them, but don't have any pre-existing knowledge.

Quoted 3 lines

A lot of times, paraphrasing excerpts to reflect the same info in a more compact format will help you keep audience on track (a little longer at least).

Yes, thank you for your help in spotting these kinds of problems. I've been too close to the code for years, which means I have severe myopia about things like "Am I confusing everyone?". :/

Speaking of which, am I confusing everyone?

--D

#8 lore raw

PATCH 2/14 xfs: document the general theory underlying online fsck design

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the second chapter of the online fsck design documentation. This covers the general theory underlying how online fsck works.

---
 .../filesystems/xfs-online-fsck-design.rst         |  366 ++++++++++++++++++++
 1 file changed, 366 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +366 -0
index 25717ebb5f80..a03a7b9f0250 100644
@@ -197,3 +197,369 @@ metadata to enable targeted checking and repair operations while the system
 is running.
 This capability will be coupled to automatic system management so that
 autonomous self-healing of XFS maximizes service availability.
+
+2. Theory of Operation
+======================
+
+Because it is necessary for online fsck to lock and scan live metadata objects,
+online fsck consists of three separate code components.
+The first is the userspace driver program ``xfs_scrub``, which is responsible
+for identifying individual metadata items, scheduling work items for them,
+reacting to the outcomes appropriately, and reporting results to the system
+administrator.
+The second and third are in the kernel, which implements functions to check
+and repair each type of online fsck work item.
+
++------------------------------------------------------------------+
+| **Note**:                                                        |
++------------------------------------------------------------------+
+| For brevity, this document shortens the phrase "online fsck work |
+| item" to "scrub item".                                           |
++------------------------------------------------------------------+
+
+Scrub item types are delineated in a manner consistent with the Unix design
+philosophy, which is to say that each item should handle one aspect of a
+metadata structure, and handle it well.
+
+Scope
+-----
+
+In principle, online fsck should be able to check and to repair everything that
+the offline fsck program can handle.
+However, the adjective *online* brings with it the limitation that online fsck
+cannot deal with anything that prevents the filesystem from going on line, i.e.
+mounting.
+This limitation means that maintenance of the offline fsck tool will continue.
+A second limitation of online fsck is that it must follow the same resource
+sharing and lock acquisition rules as the regular filesystem.
+This means that scrub cannot take *any* shortcuts to save time, because doing
+so could lead to concurrency problems.
+In other words, online fsck will never be able to fix 100% of the
+inconsistencies that offline fsck can repair, and a complete run of online fsck
+may take longer.
+However, both of these limitations are acceptable tradeoffs to satisfy the
+different motivations of online fsck, which are to **minimize system downtime**
+and to **increase predictability of operation**.
+
+.. _scrubphases:
+
+Phases of Work
+--------------
+
+The userspace driver program ``xfs_scrub`` splits the work of checking and
+repairing an entire filesystem into seven phases.
+Each phase concentrates on checking specific types of scrub items and depends
+on the success of all previous phases.
+The seven phases are as follows:
+
+1. Collect geometry information about the mounted filesystem and computer,
+   discover the online fsck capabilities of the kernel, and open the
+   underlying storage devices.
+
+2. Check allocation group metadata, all realtime volume metadata, and all quota
+   files.
+   Each metadata structure is scheduled as a separate scrub item.
+   If corruption is found in the inode header or inode btree and ``xfs_scrub``
+   is permitted to perform repairs, then those scrub items are repaired to
+   prepare for phase 3.
+   Repairs are implemented by resubmitting the scrub item to the kernel with
+   the repair flag enabled; this is discussed in the next section.
+   Optimizations and all other repairs are deferred to phase 4.
+
+3. Check all metadata of every file in the filesystem.
+   Each metadata structure is also scheduled as a separate scrub item.
+   If repairs are needed, ``xfs_scrub`` is permitted to perform repairs,
+   and there were no problems detected during phase 2, then those scrub items
+   are repaired.
+   Optimizations and unsuccessful repairs are deferred to phase 4.
+
+4. All remaining repairs and scheduled optimizations are performed during this
+   phase, if the caller permits them.
+   Before starting repairs, the summary counters are checked and any necessary
+   repairs are performed so that subsequent repairs will not fail the resource
+   reservation step due to wildly incorrect summary counters.
+   Unsuccesful repairs are requeued as long as forward progress on repairs is
+   made somewhere in the filesystem.
+   Free space in the filesystem is trimmed at the end of phase 4 if the
+   filesystem is clean.
+
+5. By the start of this phase, all primary and secondary filesystem metadata
+   must be correct.
+   Summary counters such as the free space counts and quota resource counts
+   are checked and corrected.
+   Directory entry names and extended attribute names are checked for
+   suspicious entries such as control characters or confusing Unicode sequences
+   appearing in names.
+
+6. If the caller asks for a media scan, read all allocated and written data
+   file extents in the filesystem.
+   The ability to use hardware-assisted data file integrity checking is new
+   to online fsck; neither of the previous tools have this capability.
+   If media errors occur, they will be mapped to the owning files and reported.
+
+7. Re-check the summary counters and presents the caller with a summary of
+   space usage and file counts.
+
+Steps for Each Scrub Item
+-------------------------
+
+The kernel scrub code uses a three-step strategy for checking and repairing
+the one aspect of a metadata object represented by a scrub item:
+
+1. The scrub item of interest is checked for corruptions; opportunities for
+   optimization; and for values that are directly controlled by the system
+   administrator but look suspicious.
+   If the item is not corrupt or does not need optimization, resource are
+   released and the positive scan results are returned to userspace.
+   If the item is corrupt or could be optimized but the caller does not permit
+   this, resources are released and the negative scan results are returned to
+   userspace.
+   Otherwise, the kernel moves on to the second step.
+
+2. The repair function is called to rebuild the data structure.
+   Repair functions generally choose rebuild a structure from other metadata
+   rather than try to salvage the existing structure.
+   If the repair fails, the scan results from the first step are returned to
+   userspace.
+   Otherwise, the kernel moves on to the third step.
+
+3. In the third step, the kernel runs the same checks over the new metadata
+   item to assess the efficacy of the repairs.
+   The results of the reassessment are returned to userspace.
+
+Classification of Metadata
+--------------------------
+
+Each type of metadata object (and therefore each type of scrub item) is
+classified as follows:
+
+Primary Metadata
+````````````````
+
+Metadata structures in this category should be most familiar to filesystem
+users either because they are directly created by the user or they index
+objects created by the user
+Most filesystem objects fall into this class.
+Resource and lock acquisition for scrub code follows the same order as regular
+filesystem accesses.
+
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or an inode) that
+owns the item being scrubbed is locked to guard against concurrent updates.
+The check function examines every record associated with the type for obvious
+errors and cross-references healthy records against other metadata to look for
+inconsistencies.
+Repairs for this class of scrub item are simple, since the repair function
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all the
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and commits it
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully reaped.
+
+Because ``xfs_scrub`` locks a primary object for the duration of the repair,
+this is effectively an offline repair operation performed on a subset of the
+filesystem.
+This minimizes the complexity of the repair code because it is not necessary to
+handle concurrent updates from other threads, nor is it necessary to access
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and programs
+trying to access the damaged structure will be blocked until repairs complete.
+The only infrastructure needed by the repair code are the staging area for
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair holds is clear:
+targeted work on individual shards of the filesystem avoids total loss of
+service.
+
+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,
+*Extending Database Technology*, pp. 293-309, 1992.
+
+Most primary metadata repair functions stage their intermediate results in an
+in-memory array prior to formatting the new ondisk structure, which is very
+similar to the list-based algorithm discussed in section 2.3 ("List-Based
+Algorithms") of Srinivasan.
+However, any data structure builder that maintains a resource lock for the
+duration of the repair is *always* an offline algorithm.
+
+Secondary Metadata
+``````````````````
+
+Metadata structures in this category reflect records found in primary metadata,
+but are only needed for online fsck or for reorganization of the filesystem.
+Resource and lock acquisition for scrub code do not follow the same order as
+regular filesystem accesses, and may involve full filesystem scans.
+
+Secondary metadata objects are difficult for scrub to process, because scrub
+attaches to the secondary object but needs to check primary metadata, which
+runs counter to the usual order of resource acquisition.
+Check functions can be limited in scope to reduce runtime.
+Repairs, however, require a full scan of primary metadata, which can take a
+long time to complete.
+Under these conditions, ``xfs_scrub`` cannot lock resources for the entire
+duration of the repair.
+
+Instead, repair functions set up an in-memory staging structure to store
+observations.
+Depending on the requirements of the specific repair function, the staging
+index can have the same format as the ondisk structure, or it can have a design
+specific to that repair function.
+The next step is to release all locks and start the filesystem scan.
+When the repair scanner needs to record an observation, the staging data are
+locked long enough to apply the update.
+Simultaneously, the repair function hooks relevant parts of the filesystem to
+apply updates to the staging data if the the update pertains to an object that
+has already been scanned by the index builder.
+Once the scan is done, the owning object is re-locked, the live data is used to
+write a new ondisk structure, and the repairs are committed atomically.
+The hooks are disabled and the staging staging area is freed.
+Finally, the storage from the old data structure are carefully reaped.
+
+Introducing concurrency helps online repair avoid various locking problems, but
+comes at a high cost to code complexity.
+Live filesystem code has to be hooked so that the repair function can observe
+updates in progress.
+The staging area has to become a fully functional parallel structure so that
+updates can be merged from the hooks.
+Finally, the hook, the filesystem scan, and the inode locking model must be
+sufficiently well integrated that a hook event can decide if a given update
+should be applied to the staging structure.
+
+In theory, the scrub implementation could apply these same techniques for
+primary metadata, but doing so would make it massively more complex and less
+performant.
+Programs attempting to access the damaged structures are not blocked from
+operation, which may cause application failure or an unplanned filesystem
+shutdown.
+
+Inspiration for the secondary metadata repair strategy was drawn from section
+2.4 of Srinivasan above, and sections 2 ("NSF: Inded Build Without Side-File")
+and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms for
+Creating Indexes for Very Large Tables Without Quiescing Updates"
+<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.
+
+The sidecar index mentioned above bears some resemblance to the side file
+method mentioned in Srinivasan and Mohan.
+Their method consists of an index builder that extracts relevant record data to
+build the new structure as quickly as possible; and an auxiliary structure that
+captures all updates that would be committed to the index by other threads were
+the new index already online.
+After the index building scan finishes, the updates recorded in the side file
+are applied to the new index.
+To avoid conflicts between the index builder and other writer threads, the
+builder maintains a publicly visible cursor that tracks the progress of the
+scan through the record space.
+To avoid duplication of work between the side file and the index builder, side
+file updates are elided when the record ID for the update is greater than the
+cursor position within the record ID space.
+
+To minimize changes to the rest of the codebase, XFS online repair keeps the
+replacement index hidden until it's completely ready to go.
+In other words, there is no attempt to expose the keyspace of the new index
+while repair is running.
+The complexity of such an approach would be very high and perhaps more
+appropriate to building *new* indices.
+
+**Question**: Can the full scan and live update code used to facilitate a
+repair also be used to implement a comprehensive check?
+
+*Answer*: Probably, though this has not been yet been studied.
+
+Summary Information
+```````````````````
+
+Metadata structures in this last category summarize the contents of primary
+metadata records.
+These are often used to speed up resource usage queries, and are many times
+smaller than the primary metadata which they represent.
+Check and repair both require full filesystem scans, but resource and lock
+acquisition follow the same paths as regular filesystem accesses.
+
+The superblock summary counters have special requirements due to the underlying
+implementation of the incore counters, and will be treated separately.
+Check and repair of the other types of summary counters (quota resource counts
+and file link counts) employ the same filesystem scanning and hooking
+techniques as outlined above, but because the underlying data are sets of
+integer counters, the staging data need not be a fully functional mirror of the
+ondisk structure.
+
+Inspiration for quota and file link count repair strategies were drawn from
+sections 2.12 ("Online Index Operations") through 2.14 ("Incremental View
+Maintenace") of G.  Graefe, `"Concurrent Queries and Updates in Summary Views
+and Their Indexes"
+<http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf>`_, 2011.
+
+Since quotas are non-negative integer counts of resource usage, online
+quotacheck can use the incremental view deltas described in section 2.14 to
+track pending changes to the block and inode usage counts in each transaction,
+and commit those changes to a dquot side file when the transaction commits.
+Delta tracking is necessary for dquots because the index builder scans inodes,
+whereas the data structure being rebuilt is an index of dquots.
+Link count checking combines the view deltas and commit step into one because
+it sets attributes of the objects being scanned instead of writing them to a
+separate data structure.
+Each online fsck function will be discussed as case studies later in this
+document.
+
+Risk Management
+---------------
+
+During the development of online fsck, several risk factors were identified
+that may make the feature unsuitable for certain distributors and users.
+Steps can be taken to mitigate or eliminate those risks, though at a cost to
+functionality.
+
+- **Decreased performance**: Adding metadata indices to the filesystem
+  increases the time cost of persisting changes to disk, and the reverse space
+  mapping and directory parent pointers are no exception.
+  System administrators who require the maximum performance can disable the
+  reverse mapping features at format time, though this choice dramatically
+  reduces the ability of online fsck to find inconsistencies and repair them.
+
+- **Incorrect repairs**: As with all software, there might be defects in the
+  software that result in incorrect repairs being written to the filesystem.
+  Systematic fuzz testing (detailed in the next section) is employed by the
+  authors to find bugs early, but it might not catch everything.
+  The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
+  and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose not to
+  accept this risk.
+  The xfsprogs build system has a configure option (``--enable-scrub=no``) that
+  disables building of the ``xfs_scrub`` binary, though this is not a risk
+  mitigation if the kernel functionality remains enabled.
+
+- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
+  repairable.
+  If the keyspaces of several metadata indices overlap in some manner but a
+  coherent narrative cannot be formed from records collected, then the repair
+  fails.
+  To reduce the chance that a repair will fail with a dirty transaction and
+  render the filesystem unusable, the online repair functions have been
+  designed to stage and validate all new records before committing the new
+  structure.
+
+- **Misbehavior**: Online fsck requires many privileges -- raw IO to block
+  devices, opening files by handle, ignoring Unix discretionary access control,
+  and the ability to perform administrative changes.
+  Running this automatically in the background scares people, so the systemd
+  background service is configured to run with only the privileges required.
+  Obviously, this cannot address certain problems like the kernel crashing or
+  deadlocking, but it should be sufficient to prevent the scrub process from
+  escaping and reconfiguring the system.
+  The cron job does not have this protection.
+
+- **Fuzz Kiddiez**: There are many people now who seem to think that running
+  automated fuzz testing of ondisk artifacts to find mischevious behavior and
+  spraying exploit code onto the public mailing list for instant zero-day
+  disclosure is somehow of some social benefit.
+  In the view of this author, the benefit is realized only when the fuzz
+  operators help to **fix** the flaws, but this opinion apparently is not
+  widely shared among security "researchers".
+  The XFS maintainers' continuing ability to manage these events presents an
+  ongoing risk to the stability of the development process.
+  Automated testing should front-load some of the risk while the feature is
+  considered EXPERIMENTAL.
+
+Many of these risks are inherent to software programming.
+Despite this, it is hoped that this new functionality will prove useful in
+reducing unexpected downtime.
#9 lore raw

Re: [PATCH 02/14] xfs: document the general theory underlying online fsck design

Quoted 61 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the second chapter of the online fsck design documentation. This covers the general theory underlying how online fsck works.

---
 .../filesystems/xfs-online-fsck-design.rst         |  366
++++++++++++++++++++

 1 file changed, 366 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 25717ebb5f80..a03a7b9f0250 100644
@@ -197,3 +197,369 @@ metadata to enable targeted checking and repair

operations while the system
 is running.
 This capability will be coupled to automatic system management so that
 autonomous self-healing of XFS maximizes service availability.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+2. Theory of Operation
+======================
+
+Because it is necessary for online fsck to lock and scan live

metadata objects,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+online fsck consists of three separate code components.
+The first is the userspace driver program ``xfs_scrub``, which is

responsible
+for identifying individual metadata items, scheduling work items for them,
+reacting to the outcomes appropriately, and reporting results to the system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+administrator.
+The second and third are in the kernel, which implements functions

to check

Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+and repair each type of online fsck work item.
+
++------------------------------------------------------------------+
+| **Note**:                                                        |
++------------------------------------------------------------------+
+| For brevity, this document shortens the phrase "online fsck work |
+| item" to "scrub item".                                           |
++------------------------------------------------------------------+
+
+Scrub item types are delineated in a manner consistent with the Unix

design
+philosophy, which is to say that each item should handle one aspect of a

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+metadata structure, and handle it well.
+
+Scope
+-----
+
+In principle, online fsck should be able to check and to repair

everything that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the offline fsck program can handle.
+However, the adjective *online* brings with it the limitation that
Quoted 4 lines

online fsck
+cannot deal with anything that prevents the filesystem from going on line, i.e.
+mounting.

Are there really any other operations that do that other than mount? I think this reads cleaner:

By definition, online fsck can only check and repair an online filesystem. It cannot check mounting operations which start from an offline state.

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system

Quoted 10 lines

+This limitation means that maintenance of the offline fsck tool will continue.
+A second limitation of online fsck is that it must follow the same resource

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+sharing and lock acquisition rules as the regular filesystem.
+This means that scrub cannot take *any* shortcuts to save time,

because doing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+so could lead to concurrency problems.
+In other words, online fsck will never be able to fix 100% of the
+inconsistencies that offline fsck can repair, 

Hmm, what inconsistencies cannot repaired as a result of the "no shortcut" rule? I'm all for keeping things short and to the point, but since this section is about scope, I'd give it at least a brief bullet list

Quoted 22 lines

and a complete run of online fsck

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+may take longer.
+However, both of these limitations are acceptable tradeoffs to

satisfy the
+different motivations of online fsck, which are to **minimize system downtime**

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+and to **increase predictability of operation**.
+
+.. _scrubphases:
+
+Phases of Work
+--------------
+
+The userspace driver program ``xfs_scrub`` splits the work of

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repairing an entire filesystem into seven phases.
+Each phase concentrates on checking specific types of scrub items

and depends

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+on the success of all previous phases.
+The seven phases are as follows:
+
+1. Collect geometry information about the mounted filesystem and
Quoted 8 lines

computer,
+   discover the online fsck capabilities of the kernel, and open the +   underlying storage devices.
+
+2. Check allocation group metadata, all realtime volume metadata, and all quota
+   files.
+   Each metadata structure is scheduled as a separate scrub item.

Like an intent item?

Quoted 7 lines

+   If corruption is found in the inode header or inode btree and ``xfs_scrub``
+   is permitted to perform repairs, then those scrub items are repaired to
+   prepare for phase 3.
+   Repairs are implemented by resubmitting the scrub item to the kernel with

If I'm understanding this correctly:
Repairs are implemented as intent items that are queued and committed just as any filesystem operation.

?

Quoted 2 lines

+   the repair flag enabled; this is discussed in the next section. +   Optimizations and all other repairs are deferred to phase 4.

I guess I'll come back to it.

Quoted 6 lines

+
+3. Check all metadata of every file in the filesystem. +   Each metadata structure is also scheduled as a separate scrub item.
+   If repairs are needed, ``xfs_scrub`` is permitted to perform repairs,

If repairs are needed and ``xfs_scrub`` is permitted

?

Quoted 9 lines

+   and there were no problems detected during phase 2, then those scrub items
+   are repaired.
+   Optimizations and unsuccessful repairs are deferred to phase 4. +
+4. All remaining repairs and scheduled optimizations are performed during this
+   phase, if the caller permits them.
+   Before starting repairs, the summary counters are checked and any

Did we talk about summary counters yet? Maybe worth a blub. Otherwise this may not make sense with out skipping ahead or into the code

Quoted 6 lines

necessary
+   repairs are performed so that subsequent repairs will not fail the resource
+   reservation step due to wildly incorrect summary counters. +   Unsuccesful repairs are requeued as long as forward progress on repairs is

Quoted 8 lines

+   made somewhere in the filesystem.
+   Free space in the filesystem is trimmed at the end of phase 4 if the
+   filesystem is clean.
+
+5. By the start of this phase, all primary and secondary filesystem metadata
+   must be correct.

I think maybe the definitions of primary and secondary metadata should move up before the phases section. Otherwise the reader has to skip ahead to know what that means.

Quoted 70 lines

+   Summary counters such as the free space counts and quota resource counts
+   are checked and corrected.
+   Directory entry names and extended attribute names are checked for
+   suspicious entries such as control characters or confusing Unicode sequences
+   appearing in names.
+
+6. If the caller asks for a media scan, read all allocated and written data
+   file extents in the filesystem.
+   The ability to use hardware-assisted data file integrity checking is new
+   to online fsck; neither of the previous tools have this capability.
+   If media errors occur, they will be mapped to the owning files and reported.
+
+7. Re-check the summary counters and presents the caller with a summary of
+   space usage and file counts.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Steps for Each Scrub Item
+-------------------------
+
+The kernel scrub code uses a three-step strategy for checking and

repairing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the one aspect of a metadata object represented by a scrub item:
+
+1. The scrub item of intere

st is checked for corruptions; opportunities for
+   optimization; and for values that are directly controlled by the system
+   administrator but look suspicious.
+   If the item is not corrupt or does not need optimization, resource are
+   released and the positive scan results are returned to userspace. +   If the item is corrupt or could be optimized but the caller does not permit
+   this, resources are released and the negative scan results are returned to
+   userspace.
+   Otherwise, the kernel moves on to the second step.
+
+2. The repair function is called to rebuild the data structure. +   Repair functions generally choose rebuild a structure from other metadata
+   rather than try to salvage the existing structure.
+   If the repair fails, the scan results from the first step are returned to
+   userspace.
+   Otherwise, the kernel moves on to the third step.
+
+3. In the third step, the kernel runs the same checks over the new metadata
+   item to assess the efficacy of the repairs.
+   The results of the reassessment are returned to userspace.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Classification of Metadata
+--------------------------
+
+Each type of metadata object (and therefore each type of scrub item)

is

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+classified as follows:
+
+Primary Metadata
+````````````````
+
+Metadata structures in this category should be most familiar to
Quoted 4 lines

filesystem
+users either because they are directly created by the user or they index
+objects created by the user

I think I would just jump straight into a brief list. The above is a bit vague, and documentation that tells you you should already know what it is, doesnt add much. Again, I think too much poetry might be why you're having a hard time getting responses.

Quoted 1 line

+Most filesystem objects fall into this class.

Most filesystem objects created by users fall into this class, such as inode, directories, allocation groups and so on.

Quoted 3 lines

+Resource and lock acquisition for scrub code follows the same order as regular
+filesystem accesses.

Lock acquisition for these resources will follow the same order for scrub as a regular filesystem access.

Quoted 48 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or an

inode) that
+owns the item being scrubbed is locked to guard against concurrent updates.
+The check function examines every record associated with the type for obvious
+errors and cross-references healthy records against other metadata to look for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistencies.
+Repairs for this class of scrub item are simple, since the repair

function

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all

the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and

commits it

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Because ``xfs_scrub`` locks a primary object for the duration of the

repair,
+this is effectively an offline repair operation performed on a subset of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem.
+This minimizes the complexity of the repair code because it is not

necessary to
+handle concurrent updates from other threads, nor is it necessary to access

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and

programs
+trying to access the damaged structure will be blocked until repairs complete.
+The only infrastructure needed by the repair code are the staging area for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair holds is

clear:
+targeted work on individual shards of the filesystem avoids total loss of

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+service.
+
+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index
Quoted 2 lines

Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,

Hmm, this article is not displaying for me. If the link is abandoned, probably there's not much need to keep it around

Quoted 17 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+*Extending Database Technology*, pp. 293-309, 1992.
+
+Most primary metadata repair functions stage their intermediate

results in an
+in-memory array prior to formatting the new ondisk structure, which is very
+similar to the list-based algorithm discussed in section 2.3 ("List- Based

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Algorithms") of Srinivasan.
+However, any data structure builder that maintains a resource lock

for the

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+duration of the repair is *always* an offline algorithm.
+
+Secondary Metadata
+``````````````````
+
+Metadata structures in this category reflect records found in
Quoted 1 line

primary metadata,

such as rmap and parent pointer attributes. But they are only needed...

?

Quoted 11 lines

+but are only needed for online fsck or for reorganization of the filesystem.
+Resource and lock acquisition for scrub code do not follow the same order as

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+regular filesystem accesses, and may involve full filesystem scans.
+
+Secondary metadata objects are difficult for scrub to process,

because scrub
+attaches to the secondary object but needs to check primary metadata, which
+runs counter to the usual order of resource acquisition.

bummer :-(

Quoted 15 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Check functions can be limited in scope to reduce runtime.
+Repairs, however, require a full scan of primary metadata, which can

take a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+long time to complete.
+Under these conditions, ``xfs_scrub`` cannot lock resources for the

entire

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+duration of the repair.
+
+Instead, repair functions set up an in-memory staging structure to

store

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations.
+Depending on the requirements of the specific repair function, the

staging

Quoted 3 lines

+index can have the same format as the ondisk structure, or it can have a design
+specific to that repair function.

...will have either the same format as the ondisk structure or a structure specific to the repair function.

Quoted 9 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The next step is to release all locks and start the filesystem scan.
+When the repair scanner needs to record an observation, the staging

data are

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+locked long enough to apply the update.
+Simultaneously, the repair function hooks relevant parts of the

filesystem to
+apply updates to the staging data if the the update pertains to an object that
+has already been scanned by the index builder.

While a scan is in progress, function hooks are used to apply filesystem updates to both the object and the staging data if the object has already been scanned.

?

Quoted 35 lines

+Once the scan is done, the owning object is re-locked, the live data is used to
+write a new ondisk structure, and the repairs are committed atomically.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The hooks are disabled and the staging staging area is freed.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Introducing concurrency helps online repair avoid various locking

problems, but

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+comes at a high cost to code complexity.
+Live filesystem code has to be hooked so that the repair function

can observe

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates in progress.
+The staging area has to become a fully functional parallel structure

so that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates can be merged from the hooks.
+Finally, the hook, the filesystem scan, and the inode locking model

must be
+sufficiently well integrated that a hook event can decide if a given update

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+should be applied to the staging structure.
+
+In theory, the scrub implementation could apply these same

techniques for
+primary metadata, but doing so would make it massively more complex and less

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+performant.
+Programs attempting to access the damaged structures are not blocked

from
+operation, which may cause application failure or an unplanned filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+shutdown.
+
+Inspiration for the secondary metadata repair strategy was drawn
Quoted 7 lines

from section
+2.4 of Srinivasan above, and sections 2 ("NSF: Inded Build Without Side-File")
+and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms for
+Creating Indexes for Very Large Tables Without Quiescing Updates" +<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.

This one works

Quoted 37 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The sidecar index mentioned above bears some resemblance to the side

file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+method mentioned in Srinivasan and Mohan.
+Their method consists of an index builder that extracts relevant

record data to
+build the new structure as quickly as possible; and an auxiliary structure that
+captures all updates that would be committed to the index by other threads were

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the new index already online.
+After the index building scan finishes, the updates recorded in the

side file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+are applied to the new index.
+To avoid conflicts between the index builder and other writer

threads, the
+builder maintains a publicly visible cursor that tracks the progress of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scan through the record space.
+To avoid duplication of work between the side file and the index

builder, side
+file updates are elided when the record ID for the update is greater than the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+cursor position within the record ID space.
+
+To minimize changes to the rest of the codebase, XFS online repair

keeps the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+replacement index hidden until it's completely ready to go.
+In other words, there is no attempt to expose the keyspace of the

new index

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+while repair is running.
+The complexity of such an approach would be very high and perhaps

more

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate to building *new* indices.
+
+**Question**: Can the full scan and live update code used to

facilitate a

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 467

Quoted 3 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair also be used to implement a comprehensive check?
+
+*Answer*: Probably, though this has not been yet been studied.

I kinda feel like discussion Q&As need to be wrapped up before we can call things done. If this is all there was to the answer, then lets clean out the discussion notes.

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 471

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Summary Information
+```````````````````
+

Oh, perhaps this section could move up with the other metadata definitions. That way the reader already has an idea of what these terms are referring to before we get into how they are used during the phases.

Quoted 26 lines

+Metadata structures in this last category summarize the contents of primary

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata records.
+These are often used to speed up resource usage queries, and are

many times

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+smaller than the primary metadata which they represent.
+Check and repair both require full filesystem scans, but resource

and lock

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+acquisition follow the same paths as regular filesystem accesses.
+
+The superblock summary counters have special requirements due to the

underlying
+implementation of the incore counters, and will be treated separately.
+Check and repair of the other types of summary counters (quota resource counts
+and file link counts) employ the same filesystem scanning and hooking
+techniques as outlined above, but because the underlying data are sets of
+integer counters, the staging data need not be a fully functional mirror of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+ondisk structure.
+
+Inspiration for quota and file link count repair strategies were

drawn from

Quoted 8 lines

+sections 2.12 ("Online Index Operations") through 2.14 ("Incremental View
+Maintenace") of G.  Graefe, `"Concurrent Queries and Updates in Summary Views
+and Their Indexes"
+<
http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf>`
_, 2011.

I wonder if these citations would do better as foot notes? Just to kinda keep the body of the document tidy and flowing well.

Quoted 86 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Since quotas are non-negative integer counts of resource usage,

online
+quotacheck can use the incremental view deltas described in section 2.14 to
+track pending changes to the block and inode usage counts in each transaction,
+and commit those changes to a dquot side file when the transaction commits.
+Delta tracking is necessary for dquots because the index builder scans inodes,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+whereas the data structure being rebuilt is an index of dquots.
+Link count checking combines the view deltas and commit step into

one because
+it sets attributes of the objects being scanned instead of writing them to a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+separate data structure.
+Each online fsck function will be discussed as case studies later in

this

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+document.
+
+Risk Management
+---------------
+
+During the development of online fsck, several risk factors were

identified
+that may make the feature unsuitable for certain distributors and users.
+Steps can be taken to mitigate or eliminate those risks, though at a cost to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+functionality.
+
+- **Decreased performance**: Adding metadata indices to the

filesystem
+  increases the time cost of persisting changes to disk, and the reverse space
+  mapping and directory parent pointers are no exception. +  System administrators who require the maximum performance can disable the
+  reverse mapping features at format time, though this choice dramatically
+  reduces the ability of online fsck to find inconsistencies and repair them.
+
+- **Incorrect repairs**: As with all software, there might be defects in the
+  software that result in incorrect repairs being written to the filesystem.
+  Systematic fuzz testing (detailed in the next section) is employed by the
+  authors to find bugs early, but it might not catch everything. +  The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
+  and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose not to
+  accept this risk.
+  The xfsprogs build system has a configure option (``--enable- scrub=no``) that
+  disables building of the ``xfs_scrub`` binary, though this is not a risk
+  mitigation if the kernel functionality remains enabled. +
+- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
+  repairable.
+  If the keyspaces of several metadata indices overlap in some manner but a
+  coherent narrative cannot be formed from records collected, then the repair
+  fails.
+  To reduce the chance that a repair will fail with a dirty transaction and
+  render the filesystem unusable, the online repair functions have been
+  designed to stage and validate all new records before committing the new
+  structure.
+
+- **Misbehavior**: Online fsck requires many privileges -- raw IO to block
+  devices, opening files by handle, ignoring Unix discretionary access control,
+  and the ability to perform administrative changes.
+  Running this automatically in the background scares people, so the systemd
+  background service is configured to run with only the privileges

Quoted 8 lines

required.
+  Obviously, this cannot address certain problems like the kernel crashing or
+  deadlocking, but it should be sufficient to prevent the scrub process from
+  escaping and reconfiguring the system.
+  The cron job does not have this protection.
+

I think the fuzz part is one I would consider letting go. All features need to go through a period of stabilizing, and we cant really control how some people respond to it, so I don't think this part adds much. I think the document would do well to be trimmed where it can so as to stay more focused

Quoted 21 lines

+- **Fuzz Kiddiez**: There are many people now who seem to think that running
+  automated fuzz testing of ondisk artifacts to find mischevious behavior and
+  spraying exploit code onto the public mailing list for instant zero-day
+  disclosure is somehow of some social benefit.
+  In the view of this author, the benefit is realized only when the fuzz
+  operators help to **fix** the flaws, but this opinion apparently is not
+  widely shared among security "researchers".
+  The XFS maintainers' continuing ability to manage these events presents an
+  ongoing risk to the stability of the development process. +  Automated testing should front-load some of the risk while the feature is
+  considered EXPERIMENTAL.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Many of these risks are inherent to software programming.
+Despite this, it is hoped that this new functionality will prove
Quoted 3 lines

useful in
+reducing unexpected downtime.

Paraphrasing and reorganizing suggestions aside, I think it looks pretty good

Allison

#10 lore raw

Re: [PATCH 02/14] xfs: document the general theory underlying online fsck design

Quoted 59 lines · Allison Henderson
Quoted 58 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the second chapter of the online fsck design documentation. This covers the general theory underlying how online fsck works.

---
 .../filesystems/xfs-online-fsck-design.rst         |  366
++++++++++++++++++++

 1 file changed, 366 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 25717ebb5f80..a03a7b9f0250 100644
@@ -197,3 +197,369 @@ metadata to enable targeted checking and repair

operations while the system
 is running.
 This capability will be coupled to automatic system management so that
 autonomous self-healing of XFS maximizes service availability.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+2. Theory of Operation
+======================
+
+Because it is necessary for online fsck to lock and scan live

metadata objects,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+online fsck consists of three separate code components.
+The first is the userspace driver program ``xfs_scrub``, which is

responsible
+for identifying individual metadata items, scheduling work items for them,
+reacting to the outcomes appropriately, and reporting results to the system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+administrator.
+The second and third are in the kernel, which implements functions

to check

Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+and repair each type of online fsck work item.
+
++------------------------------------------------------------------+
+| **Note**:                                                        |
++------------------------------------------------------------------+
+| For brevity, this document shortens the phrase "online fsck work |
+| item" to "scrub item".                                           |
++------------------------------------------------------------------+
+
+Scrub item types are delineated in a manner consistent with the Unix

design
+philosophy, which is to say that each item should handle one aspect of a

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+metadata structure, and handle it well.
+
+Scope
+-----
+
+In principle, online fsck should be able to check and to repair
Quoted 8 lines
Quoted 7 lines

everything that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the offline fsck program can handle.
+However, the adjective *online* brings with it the limitation that

online fsck
+cannot deal with anything that prevents the filesystem from going on line, i.e.
+mounting.

Are there really any other operations that do that other than mount?

No.

Quoted 5 lines

I think this reads cleaner:

By definition, online fsck can only check and repair an online filesystem. It cannot check mounting operations which start from an offline state.

Now that I think about this some more, this whole sentence doesn't make sense. xfs_scrub can *definitely* detect and fix latent errors that would prevent the /next/ mount from succeeding. It's only the fuzz test suite that stumbles over this, and only because xfs_db cannot fuzz mounted filesystems.

"However, online fsck cannot be running 100% of the time, which means that latent errors may creep in after a scrub completes.
If these errors cause the next mount to fail, offline fsck is the only solution."

Quoted 6 lines
Quoted 6 lines

+This limitation means that maintenance of the offline fsck tool will continue.
+A second limitation of online fsck is that it must follow the same resource

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+sharing and lock acquisition rules as the regular filesystem.
+This means that scrub cannot take *any* shortcuts to save time,
Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system

Quoted 4 lines

because doing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+so could lead to concurrency problems.
+In other words, online fsck will never be able to fix 100% of the
+inconsistencies that offline fsck can repair, 

Hmm, what inconsistencies cannot repaired as a result of the "no shortcut" rule? I'm all for keeping things short and to the point, but since this section is about scope, I'd give it at least a brief bullet list

Hmm. I can't think of any off the top of my head. Given the rewording earlier, I think it's more accurate to say:

"In other words, online fsck is not a complete replacement for offline fsck, and a complete run of online fsck may take longer than online fsck."

Quoted 23 lines
Quoted 23 lines

and a complete run of online fsck

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+may take longer.
+However, both of these limitations are acceptable tradeoffs to

satisfy the
+different motivations of online fsck, which are to **minimize system downtime**

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+and to **increase predictability of operation**.
+
+.. _scrubphases:
+
+Phases of Work
+--------------
+
+The userspace driver program ``xfs_scrub`` splits the work of

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repairing an entire filesystem into seven phases.
+Each phase concentrates on checking specific types of scrub items

and depends

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+on the success of all previous phases.
+The seven phases are as follows:
+
+1. Collect geometry information about the mounted filesystem and

computer,

Quoted 8 lines
Quoted 7 lines

+   discover the online fsck capabilities of the kernel, and open the +   underlying storage devices.
+
+2. Check allocation group metadata, all realtime volume metadata, and all quota
+   files.
+   Each metadata structure is scheduled as a separate scrub item.

Like an intent item?

No, these scrub items are struct scrub_item objects that exist solely within the userspace program code.

Quoted 7 lines
Quoted 7 lines

+   If corruption is found in the inode header or inode btree and ``xfs_scrub``
+   is permitted to perform repairs, then those scrub items are repaired to
+   prepare for phase 3.
+   Repairs are implemented by resubmitting the scrub item to the kernel with

Quoted 5 lines

If I'm understanding this correctly:
Repairs are implemented as intent items that are queued and committed just as any filesystem operation.

?

I don't want to go too deep into this prematurely, but...

xfs_scrub (the userspace program) needs to track which metadata objects have been checked and which ones need repairs. The current codebase (ab)uses struct xfs_scrub_metadata, but it's very memory inefficient. I replaced it with a new struct scrub_item that stores (a) all the handle information to identify the inode/AG/rt group/whatever; and (b) the state of all the checks that can be applied to that item:

struct scrub_item {
	/*
	 * Information we need to call the scrub and repair ioctls.
	 * Per-AG items should set the ino/gen fields to -1; per-inode
	 * items should set sri_agno to -1; and per-fs items should set
	 * all three fields to -1.  Or use the macros below.
	 */
	__u64			sri_ino;
	__u32			sri_gen;
	__u32			sri_agno;

	/* Bitmask of scrub types that were scheduled here. */
	__u32			sri_selected;

	/* Scrub item state flags, one for each XFS_SCRUB_TYPE. */
	__u8			sri_state[XFS_SCRUB_TYPE_NR];

	/* Track scrub and repair call retries for each scrub type. */
	__u8			sri_tries[XFS_SCRUB_TYPE_NR];

	/* Were there any corruption repairs needed? */
	bool			sri_inconsistent:1;

	/* Are we revalidating after repairs? */
	bool			sri_revalidate:1;
};

The first three fields are passed to the kernel via scrub ioctl and describe a particular xfs domain (files, AGs, etc). The rest of the structure store state for each type of repair that can be performed against that domain.

IOWs, xfs_scrub uses struct scrub_item objects to generate ioctl calls to the kernel to check and repair things. The kernel reads the ioctl information, figures out what needs to be done, and then does the usual get transaction -> lock things -> make updates -> commit dance to make corrections to the fs. Those corrections include log intent items, but there's no tight coupling between log intent items and scrub_items.

Side note: The kernel repair code used to use intents to rebuild a structure, but nowadays it use the btree bulk loader code to replace btrees wholesale and in a single atomic commit. Now we use them primariliy to free preallocated space if the repair fails.

Quoted 11 lines
Quoted 2 lines

+   the repair flag enabled; this is discussed in the next section. +   Optimizations and all other repairs are deferred to phase 4.

I guess I'll come back to it.

Quoted 6 lines

+
+3. Check all metadata of every file in the filesystem. +   Each metadata structure is also scheduled as a separate scrub item.
+   If repairs are needed, ``xfs_scrub`` is permitted to perform repairs,

If repairs are needed and ``xfs_scrub`` is permitted

Fixed.

Quoted 10 lines

?

Quoted 9 lines

+   and there were no problems detected during phase 2, then those scrub items
+   are repaired.
+   Optimizations and unsuccessful repairs are deferred to phase 4. +
+4. All remaining repairs and scheduled optimizations are performed during this
+   phase, if the caller permits them.
+   Before starting repairs, the summary counters are checked and any

Quoted 2 lines

Did we talk about summary counters yet? Maybe worth a blub. Otherwise this may not make sense with out skipping ahead or into the code

Nope. I'll add that to the previous patch when I introduce primary and secondary metadata. Good catch!

"Summary metadata, as the name implies, condense information contained in primary metadata for performance reasons."

Quoted 14 lines
Quoted 6 lines

necessary
+   repairs are performed so that subsequent repairs will not fail the resource
+   reservation step due to wildly incorrect summary counters. +   Unsuccesful repairs are requeued as long as forward progress on repairs is

Quoted 8 lines

+   made somewhere in the filesystem.
+   Free space in the filesystem is trimmed at the end of phase 4 if the
+   filesystem is clean.
+
+5. By the start of this phase, all primary and secondary filesystem metadata
+   must be correct.

Quoted 3 lines

I think maybe the definitions of primary and secondary metadata should move up before the phases section. Otherwise the reader has to skip ahead to know what that means.

Yep, now primary, secondary, and summary metadata are defined in section
1. Very good comment.

Quoted 70 lines
Quoted 70 lines

+   Summary counters such as the free space counts and quota resource counts
+   are checked and corrected.
+   Directory entry names and extended attribute names are checked for
+   suspicious entries such as control characters or confusing Unicode sequences
+   appearing in names.
+
+6. If the caller asks for a media scan, read all allocated and written data
+   file extents in the filesystem.
+   The ability to use hardware-assisted data file integrity checking is new
+   to online fsck; neither of the previous tools have this capability.
+   If media errors occur, they will be mapped to the owning files and reported.
+
+7. Re-check the summary counters and presents the caller with a summary of
+   space usage and file counts.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Steps for Each Scrub Item
+-------------------------
+
+The kernel scrub code uses a three-step strategy for checking and

repairing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the one aspect of a metadata object represented by a scrub item:
+
+1. The scrub item of intere

st is checked for corruptions; opportunities for
+   optimization; and for values that are directly controlled by the system
+   administrator but look suspicious.
+   If the item is not corrupt or does not need optimization, resource are
+   released and the positive scan results are returned to userspace. +   If the item is corrupt or could be optimized but the caller does not permit
+   this, resources are released and the negative scan results are returned to
+   userspace.
+   Otherwise, the kernel moves on to the second step.
+
+2. The repair function is called to rebuild the data structure. +   Repair functions generally choose rebuild a structure from other metadata
+   rather than try to salvage the existing structure.
+   If the repair fails, the scan results from the first step are returned to
+   userspace.
+   Otherwise, the kernel moves on to the third step.
+
+3. In the third step, the kernel runs the same checks over the new metadata
+   item to assess the efficacy of the repairs.
+   The results of the reassessment are returned to userspace.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Classification of Metadata
+--------------------------
+
+Each type of metadata object (and therefore each type of scrub item)

is

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+classified as follows:
+
+Primary Metadata
+````````````````
+
+Metadata structures in this category should be most familiar to
Quoted 8 lines
Quoted 4 lines

filesystem
+users either because they are directly created by the user or they index
+objects created by the user

I think I would just jump straight into a brief list. The above is a bit vague, and documentation that tells you you should already know what it is, doesnt add much. Again, I think too much poetry might be why you're having a hard time getting responses.

Done:

- Free space and reference count information

- Inode records and indexes

- Storage mapping information for file data

- Directories

- Extended attributes

- Symbolic links

- Quota limits

- Link counts

Quoted 9 lines
Quoted 1 line

+Most filesystem objects fall into this class.

Most filesystem objects created by users fall into this class, such as inode, directories, allocation groups and so on.

Quoted 3 lines

+Resource and lock acquisition for scrub code follows the same order as regular
+filesystem accesses.

Lock acquisition for these resources will follow the same order for scrub as a regular filesystem access.

Yes, that is clearer. I think I'll phrase this more actively:

"Scrub obeys the same rules as regular filesystem accesses for resource and lock acquisition."

Quoted 44 lines
Quoted 44 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or an

inode) that
+owns the item being scrubbed is locked to guard against concurrent updates.
+The check function examines every record associated with the type for obvious
+errors and cross-references healthy records against other metadata to look for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistencies.
+Repairs for this class of scrub item are simple, since the repair

function

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all

the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and

commits it

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Because ``xfs_scrub`` locks a primary object for the duration of the

repair,
+this is effectively an offline repair operation performed on a subset of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem.
+This minimizes the complexity of the repair code because it is not

necessary to
+handle concurrent updates from other threads, nor is it necessary to access

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and

programs
+trying to access the damaged structure will be blocked until repairs complete.
+The only infrastructure needed by the repair code are the staging area for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair holds is

clear:
+targeted work on individual shards of the filesystem avoids total loss of

Quoted 8 lines
Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+service.
+
+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index

Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,

Hmm, this article is not displaying for me. If the link is abandoned, probably there's not much need to keep it around

The actual paper is not directly available through that ACM link, but the DOI is what I used to track down a paper copy(!) of that paper as published in a journal.

(In turn, that journal is "Advances in Database Technology - EDBT 1992"; I found it in the NYU library. Amazingly, they sold it to me.)

Quoted 22 lines
Quoted 17 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+*Extending Database Technology*, pp. 293-309, 1992.
+
+Most primary metadata repair functions stage their intermediate

results in an
+in-memory array prior to formatting the new ondisk structure, which is very
+similar to the list-based algorithm discussed in section 2.3 ("List- Based

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Algorithms") of Srinivasan.
+However, any data structure builder that maintains a resource lock

for the

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+duration of the repair is *always* an offline algorithm.
+
+Secondary Metadata
+``````````````````
+
+Metadata structures in this category reflect records found in
Quoted 1 line

primary metadata,

such as rmap and parent pointer attributes. But they are only needed...

Quoted 1 line

?

Euugh, this section needs some restructuring to get rid of redundant sentences. How about:

"Metadata structures in this category reflect records found in primary metadata, but are only needed for online fsck or for reorganization of the filesystem.

"Secondary metadata include:

- Reverse mapping information

- Directory parent pointers

"This class of metadata is difficult for scrub to process because scrub attaches to the secondary object but needs to check primary metadata, which runs counter to the usual order of resource acquisition. Frequently, this means that full filesystems scans are necessary to rebuild the metadata.
Check functions..."

Quoted 4 lines
Quoted 4 lines

+but are only needed for online fsck or for reorganization of the filesystem.
+Resource and lock acquisition for scrub code do not follow the same order as

Quoted 8 lines
Quoted 7 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+regular filesystem accesses, and may involve full filesystem scans.
+
+Secondary metadata objects are difficult for scrub to process,

because scrub
+attaches to the secondary object but needs to check primary metadata, which
+runs counter to the usual order of resource acquisition.

bummer :-(

Yup.

Quoted 15 lines
Quoted 13 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Check functions can be limited in scope to reduce runtime.
+Repairs, however, require a full scan of primary metadata, which can

take a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+long time to complete.
+Under these conditions, ``xfs_scrub`` cannot lock resources for the

entire

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+duration of the repair.
+
+Instead, repair functions set up an in-memory staging structure to

store

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations.
+Depending on the requirements of the specific repair function, the

staging

Quoted 5 lines
Quoted 3 lines

+index can have the same format as the ondisk structure, or it can have a design
+specific to that repair function.

...will have either the same format as the ondisk structure or a structure specific to the repair function.

Fixed.

Quoted 13 lines
Quoted 9 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The next step is to release all locks and start the filesystem scan.
+When the repair scanner needs to record an observation, the staging

data are

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+locked long enough to apply the update.
+Simultaneously, the repair function hooks relevant parts of the

filesystem to
+apply updates to the staging data if the the update pertains to an object that
+has already been scanned by the index builder.

While a scan is in progress, function hooks are used to apply filesystem updates to both the object and the staging data if the object has already been scanned.

Quoted 1 line

?

The hooks are used to apply updates to the repair staging data, but they don't apply regular filesystem updates.

The usual process runs something like this:

Lock -> update -> update -> commit

With a scan in progress, say we hook the second update. The instruction flow becomes:

Lock -> update -> update -> hook -> update staging data -> commit

Maybe something along the following would be better?

"While the filesystem scan is in progress, the repair function hooks the filesystem so that it can apply pending filesystem updates to the staging information."

Quoted 79 lines
Quoted 35 lines

+Once the scan is done, the owning object is re-locked, the live data is used to
+write a new ondisk structure, and the repairs are committed atomically.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The hooks are disabled and the staging staging area is freed.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Introducing concurrency helps online repair avoid various locking

problems, but

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+comes at a high cost to code complexity.
+Live filesystem code has to be hooked so that the repair function

can observe

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates in progress.
+The staging area has to become a fully functional parallel structure

so that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates can be merged from the hooks.
+Finally, the hook, the filesystem scan, and the inode locking model

must be
+sufficiently well integrated that a hook event can decide if a given update

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+should be applied to the staging structure.
+
+In theory, the scrub implementation could apply these same

techniques for
+primary metadata, but doing so would make it massively more complex and less

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+performant.
+Programs attempting to access the damaged structures are not blocked

from
+operation, which may cause application failure or an unplanned filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+shutdown.
+
+Inspiration for the secondary metadata repair strategy was drawn
Quoted 7 lines

from section
+2.4 of Srinivasan above, and sections 2 ("NSF: Inded Build Without Side-File")
+and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms for
+Creating Indexes for Very Large Tables Without Quiescing Updates" +<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.

This one works

Quoted 35 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The sidecar index mentioned above bears some resemblance to the side

file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+method mentioned in Srinivasan and Mohan.
+Their method consists of an index builder that extracts relevant

record data to
+build the new structure as quickly as possible; and an auxiliary structure that
+captures all updates that would be committed to the index by other threads were

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the new index already online.
+After the index building scan finishes, the updates recorded in the

side file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+are applied to the new index.
+To avoid conflicts between the index builder and other writer

threads, the
+builder maintains a publicly visible cursor that tracks the progress of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scan through the record space.
+To avoid duplication of work between the side file and the index

builder, side
+file updates are elided when the record ID for the update is greater than the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+cursor position within the record ID space.
+
+To minimize changes to the rest of the codebase, XFS online repair

keeps the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+replacement index hidden until it's completely ready to go.
+In other words, there is no attempt to expose the keyspace of the

new index

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+while repair is running.
+The complexity of such an approach would be very high and perhaps

more

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+appropriate to building *new* indices.
+
Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 467

Quoted 5 lines

+**Question**: Can the full scan and live update code used to facilitate a

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair also be used to implement a comprehensive check?
+
+*Answer*: Probably, though this has not been yet been studied.

I kinda feel like discussion Q&As need to be wrapped up before we can call things done. If this is all there was to the answer, then lets clean out the discussion notes.

Oh, the situation here is worse than that -- in theory, check would be much stronger if each scrub function employed these live scans to build a shadow copy of the metadata and then compared the records of both.

However, that increases the amount of work each scrubber has to do much higher, and the runtime of those scrubbers would go up. The other issue is that live scan hooks would have to proliferate through much more of the filesystem. That's rather more invasive to the codebase than most of fsck, so I want people to look at the usage models for the handful of scrubbers that really require it before I spread it around elsewhere. Making that kind of change isn't that difficult, but I want to merge this stuff before moving on to experimenting with improvements of that scale.

Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 471

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Summary Information
+```````````````````
+

Oh, perhaps this section could move up with the other metadata definitions. That way the reader already has an idea of what these terms are referring to before we get into how they are used during the phases.

Yeah, I think/hope this will be less of a problem now that section 1 defines all three types of metadata. The start of this section now reads:

"Metadata structures in this last category summarize the contents of primary metadata records.
These are often used to speed up resource usage queries, and are many times smaller than the primary metadata which they represent.

Examples of summary information include:

- Summary counts of free space and inodes

- File link counts from directories

- Quota resource usage counts

"Check and repair require full filesystem scans, but resource and lock acquisition follow the same paths as regular filesystem accesses."

Quoted 28 lines
Quoted 28 lines

+Metadata structures in this last category summarize the contents of primary

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata records.
+These are often used to speed up resource usage queries, and are

many times

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+smaller than the primary metadata which they represent.
+Check and repair both require full filesystem scans, but resource

and lock

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+acquisition follow the same paths as regular filesystem accesses.
+
+The superblock summary counters have special requirements due to the

underlying
+implementation of the incore counters, and will be treated separately.
+Check and repair of the other types of summary counters (quota resource counts
+and file link counts) employ the same filesystem scanning and hooking
+techniques as outlined above, but because the underlying data are sets of
+integer counters, the staging data need not be a fully functional mirror of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+ondisk structure.
+
+Inspiration for quota and file link count repair strategies were

drawn from
+sections 2.12 ("Online Index Operations") through 2.14 ("Incremental View

Quoted 8 lines
Quoted 6 lines

+Maintenace") of G.  Graefe, `"Concurrent Queries and Updates in Summary Views
+and Their Indexes"
+<
http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf>`
_, 2011.

I wonder if these citations would do better as foot notes? Just to kinda keep the body of the document tidy and flowing well.

Yes, if this were a paginated document.

Quoted 92 lines
Quoted 92 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Since quotas are non-negative integer counts of resource usage,

online
+quotacheck can use the incremental view deltas described in section 2.14 to
+track pending changes to the block and inode usage counts in each transaction,
+and commit those changes to a dquot side file when the transaction commits.
+Delta tracking is necessary for dquots because the index builder scans inodes,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+whereas the data structure being rebuilt is an index of dquots.
+Link count checking combines the view deltas and commit step into

one because
+it sets attributes of the objects being scanned instead of writing them to a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+separate data structure.
+Each online fsck function will be discussed as case studies later in

this

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+document.
+
+Risk Management
+---------------
+
+During the development of online fsck, several risk factors were

identified
+that may make the feature unsuitable for certain distributors and users.
+Steps can be taken to mitigate or eliminate those risks, though at a cost to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+functionality.
+
+- **Decreased performance**: Adding metadata indices to the

filesystem
+  increases the time cost of persisting changes to disk, and the reverse space
+  mapping and directory parent pointers are no exception. +  System administrators who require the maximum performance can disable the
+  reverse mapping features at format time, though this choice dramatically
+  reduces the ability of online fsck to find inconsistencies and repair them.
+
+- **Incorrect repairs**: As with all software, there might be defects in the
+  software that result in incorrect repairs being written to the filesystem.
+  Systematic fuzz testing (detailed in the next section) is employed by the
+  authors to find bugs early, but it might not catch everything. +  The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
+  and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose not to
+  accept this risk.
+  The xfsprogs build system has a configure option (``--enable- scrub=no``) that
+  disables building of the ``xfs_scrub`` binary, though this is not a risk
+  mitigation if the kernel functionality remains enabled. +
+- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
+  repairable.
+  If the keyspaces of several metadata indices overlap in some manner but a
+  coherent narrative cannot be formed from records collected, then the repair
+  fails.
+  To reduce the chance that a repair will fail with a dirty transaction and
+  render the filesystem unusable, the online repair functions have been
+  designed to stage and validate all new records before committing the new
+  structure.
+
+- **Misbehavior**: Online fsck requires many privileges -- raw IO to block
+  devices, opening files by handle, ignoring Unix discretionary access control,
+  and the ability to perform administrative changes.
+  Running this automatically in the background scares people, so the systemd
+  background service is configured to run with only the privileges required.
+  Obviously, this cannot address certain problems like the kernel crashing or
+  deadlocking, but it should be sufficient to prevent the scrub process from
+  escaping and reconfiguring the system.

Quoted 8 lines
Quoted 2 lines

+  The cron job does not have this protection.
+

I think the fuzz part is one I would consider letting go. All features need to go through a period of stabilizing, and we cant really control how some people respond to it, so I don't think this part adds much. I think the document would do well to be trimmed where it can so as to stay more focused

It took me a minute to realize that this comment applies to the text below it. Right?

Quoted 7 lines
Quoted 7 lines

+- **Fuzz Kiddiez**: There are many people now who seem to think that running
+  automated fuzz testing of ondisk artifacts to find mischevious behavior and
+  spraying exploit code onto the public mailing list for instant zero-day
+  disclosure is somehow of some social benefit.

I want to keep this bit because it keeps happening[2]. Some folks (huawei/alibaba?) have started to try to fix the bugs that their robots find, and kudos to them!

You might have noticed that Googlers turned their firehose back on and once again aren't doing anything to fix the problems they find. How very Googley of them.

[2] https://lwn.net/Articles/904293/

Quoted 12 lines
Quoted 12 lines

+  In the view of this author, the benefit is realized only when the fuzz
+  operators help to **fix** the flaws, but this opinion apparently is not
+  widely shared among security "researchers".
+  The XFS maintainers' continuing ability to manage these events presents an
+  ongoing risk to the stability of the development process. +  Automated testing should front-load some of the risk while the feature is
+  considered EXPERIMENTAL.
+

Quoted 8 lines
Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Many of these risks are inherent to software programming.
+Despite this, it is hoped that this new functionality will prove

useful in
+reducing unexpected downtime.

Paraphrasing and reorganizing suggestions aside, I think it looks pretty good

Ok, thank you!

--D

Quoted 1 line

Allison

#11 lore raw

Re: [PATCH 02/14] xfs: document the general theory underlying online fsck design

Quoted 54 lines · Darrick J. Wong
Quoted 44 lines · Allison Henderson
Quoted 43 lines · Darrick J. Wong
(quoted diff) +2 -0
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or an

inode) that
+owns the item being scrubbed is locked to guard against concurrent updates.
+The check function examines every record associated with the type for obvious
+errors and cross-references healthy records against other metadata to look for

(quoted diff) +2 -0
+inconsistencies.
+Repairs for this class of scrub item are simple, since the repair

function

(quoted diff) +2 -0
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all

the

(quoted diff) +2 -0
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and

commits it

(quoted diff) +2 -0
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully

reaped.

(quoted diff) +2 -0
+
+Because ``xfs_scrub`` locks a primary object for the duration of the

repair,
+this is effectively an offline repair operation performed on a subset of the

(quoted diff) +2 -0
+filesystem.
+This minimizes the complexity of the repair code because it is not

necessary to
+handle concurrent updates from other threads, nor is it necessary to access

(quoted diff) +2 -0
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and

programs
+trying to access the damaged structure will be blocked until repairs complete.
+The only infrastructure needed by the repair code are the staging area for

(quoted diff) +2 -0
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair holds is

clear:
+targeted work on individual shards of the filesystem avoids total loss of

Quoted 8 lines
Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+service.
+
+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index

Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,

Hmm, this article is not displaying for me. If the link is abandoned, probably there's not much need to keep it around

Quoted 3 lines

The actual paper is not directly available through that ACM link, but the DOI is what I used to track down a paper copy(!) of that paper as published in a journal.

PDF version here:

https://minds.wisconsin.edu/bitstream/handle/1793/59524/TR1047.pdf?sequence=1

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#12 lore raw

Re: [PATCH 02/14] xfs: document the general theory underlying online fsck design

In reply to Darrick J. Wong
Quoted 104 lines · Darrick J. Wong
Quoted 77 lines · Allison Henderson
Quoted 76 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the second chapter of the online fsck design documentation. This covers the general theory underlying how online fsck works.

---
 .../filesystems/xfs-online-fsck-design.rst         |  366
++++++++++++++++++++

 1 file changed, 366 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 25717ebb5f80..a03a7b9f0250 100644
@@ -197,3 +197,369 @@ metadata to enable targeted checking and

repair
operations while the system
 is running.
 This capability will be coupled to automatic system management so
that
 autonomous self-healing of XFS maximizes service availability.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+2. Theory of Operation
+======================
+
+Because it is necessary for online fsck to lock and scan live

metadata objects,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+online fsck consists of three separate code components.
+The first is the userspace driver program ``xfs_scrub``, which

is
responsible
+for identifying individual metadata items, scheduling work items for
them,
+reacting to the outcomes appropriately, and reporting results to the
system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+administrator.
+The second and third are in the kernel, which implements

functions
to check

Documentation/filesystems/xfs-online-fsck-design.rst +4 -1
+and repair each type of online fsck work item.
+
++---------------------------------------------------------------
---+
+|

**Note**:
|
++---------------------------------------------------------------
---+
+| For brevity, this document shortens the phrase "online fsck work |
+| item" to "scrub
item".                                           |
++---------------------------------------------------------------
---+

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Scrub item types are delineated in a manner consistent with the

Unix
design
+philosophy, which is to say that each item should handle one aspect
of a

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+metadata structure, and handle it well.
+
+Scope
+-----
+
+In principle, online fsck should be able to check and to repair

everything that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the offline fsck program can handle.
+However, the adjective *online* brings with it the limitation
Quoted 8 lines
Quoted 6 lines

that
online fsck
+cannot deal with anything that prevents the filesystem from going on
line, i.e.
+mounting.

Are there really any other operations that do that other than mount?

No.

Quoted 6 lines

I think this reads cleaner:

By definition, online fsck can only check and repair an online filesystem.  It cannot check mounting operations which start from an
offline state.

Now that I think about this some more, this whole sentence doesn't make
sense.  xfs_scrub can *definitely* detect and fix latent errors that would prevent the /next/ mount from succeeding.  It's only the fuzz test
suite that stumbles over this, and only because xfs_db cannot fuzz mounted filesystems.

Quoted 5 lines

"However, online fsck cannot be running 100% of the time, which means that latent errors may creep in after a scrub completes. If these errors cause the next mount to fail, offline fsck is the only
solution."

Sure, that sounds fair

Quoted 21 lines
Quoted 12 lines
Quoted 12 lines

+This limitation means that maintenance of the offline fsck tool will
continue.
+A second limitation of online fsck is that it must follow the same
resource

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+sharing and lock acquisition rules as the regular filesystem.
+This means that scrub cannot take *any* shortcuts to save time,

because doing

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+so could lead to concurrency problems.
+In other words, online fsck will never be able to fix 100% of
Quoted 8 lines
Quoted 2 lines

the
+inconsistencies that offline fsck can repair,

Hmm, what inconsistencies cannot repaired as a result of the "no shortcut" rule?  I'm all for keeping things short and to the point, but
since this section is about scope, I'd give it at least a brief bullet
list

Quoted 8 lines

Hmm.  I can't think of any off the top of my head.  Given the rewording
earlier, I think it's more accurate to say:

"In other words, online fsck is not a complete replacement for offline
fsck, and a complete run of online fsck may take longer than online fsck."

That makes sense

Quoted 112 lines
Quoted 29 lines
Quoted 29 lines

and a complete run of online fsck

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+may take longer.
+However, both of these limitations are acceptable tradeoffs to

satisfy the
+different motivations of online fsck, which are to **minimize system
downtime**

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+and to **increase predictability of operation**.
+
+.. _scrubphases:
+
+Phases of Work
+--------------
+
+The userspace driver program ``xfs_scrub`` splits the work of

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repairing an entire filesystem into seven phases.
+Each phase concentrates on checking specific types of scrub

items
and depends

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+on the success of all previous phases.
+The seven phases are as follows:
+
+1. Collect geometry information about the mounted filesystem and

computer,
+   discover the online fsck capabilities of the kernel, and open the
+   underlying storage devices.

Quoted 8 lines
Quoted 7 lines

+
+2. Check allocation group metadata, all realtime volume metadata,
and all quota
+   files.
+   Each metadata structure is scheduled as a separate scrub item.

Like an intent item?

No, these scrub items are struct scrub_item objects that exist solely within the userspace program code.

Quoted 7 lines
Quoted 7 lines

+   If corruption is found in the inode header or inode btree and ``xfs_scrub``
+   is permitted to perform repairs, then those scrub items are repaired to
+   prepare for phase 3.
+   Repairs are implemented by resubmitting the scrub item to the kernel with

Quoted 6 lines

If I'm understanding this correctly:
Repairs are implemented as intent items that are queued and committed
just as any filesystem operation.

?

I don't want to go too deep into this prematurely, but...

xfs_scrub (the userspace program) needs to track which metadata objects
have been checked and which ones need repairs.  The current codebase (ab)uses struct xfs_scrub_metadata, but it's very memory inefficient. I replaced it with a new struct scrub_item that stores (a) all the handle information to identify the inode/AG/rt group/whatever; and
(b)
the state of all the checks that can be applied to that item:

struct scrub_item {
        /*
         * Information we need to call the scrub and repair ioctls.
         * Per-AG items should set the ino/gen fields to -1; per- inode
         * items should set sri_agno to -1; and per-fs items should set
         * all three fields to -1.  Or use the macros below. */
        __u64                   sri_ino;
        __u32                   sri_gen;
        __u32                   sri_agno;

        /* Bitmask of scrub types that were scheduled here. */ __u32                   sri_selected;

        /* Scrub item state flags, one for each XFS_SCRUB_TYPE. */ __u8                    sri_state[XFS_SCRUB_TYPE_NR];

        /* Track scrub and repair call retries for each scrub type. */
        __u8                    sri_tries[XFS_SCRUB_TYPE_NR];

        /* Were there any corruption repairs needed? */ bool                    sri_inconsistent:1;

        /* Are we revalidating after repairs? */
        bool                    sri_revalidate:1;
};

The first three fields are passed to the kernel via scrub ioctl and describe a particular xfs domain (files, AGs, etc).  The rest of the structure store state for each type of repair that can be performed against that domain.

IOWs, xfs_scrub uses struct scrub_item objects to generate ioctl calls
to the kernel to check and repair things.  The kernel reads the ioctl information, figures out what needs to be done, and then does the usual
get transaction -> lock things -> make updates -> commit dance to make
corrections to the fs.  Those corrections include log intent items, but
there's no tight coupling between log intent items and scrub_items.

Quoted 4 lines

Side note: The kernel repair code used to use intents to rebuild a structure, but nowadays it use the btree bulk loader code to replace btrees wholesale and in a single atomic commit.  Now we use them primariliy to free preallocated space if the repair fails.

Oh ok, well how about just:

"Repairs are implemented by resubmitting the scrub item to the kernel through a designated ioctl with..."

?

Quoted 34 lines
Quoted 6 lines
Quoted 4 lines

+   the repair flag enabled; this is discussed in the next section.
+   Optimizations and all other repairs are deferred to phase 4.

I guess I'll come back to it.

Quoted 7 lines
Quoted 6 lines

+
+3. Check all metadata of every file in the filesystem. +   Each metadata structure is also scheduled as a separate scrub item.
+   If repairs are needed, ``xfs_scrub`` is permitted to perform repairs,

If repairs are needed and ``xfs_scrub`` is permitted

Fixed.

Quoted 14 lines

?

Quoted 5 lines

+   and there were no problems detected during phase 2, then those
scrub items
+   are repaired.
+   Optimizations and unsuccessful repairs are deferred to phase

Quoted 8 lines

4.
+
+4. All remaining repairs and scheduled optimizations are performed
during this
+   phase, if the caller permits them.
+   Before starting repairs, the summary counters are checked and any

Quoted 3 lines

Did we talk about summary counters yet?  Maybe worth a blub. Otherwise
this may not make sense with out skipping ahead or into the code

Quoted 7 lines

Nope.  I'll add that to the previous patch when I introduce primary and
secondary metadata.  Good catch!

"Summary metadata, as the name implies, condense information contained
in primary metadata for performance reasons."

Ok, sounds good then

Quoted 154 lines
Quoted 19 lines
Quoted 11 lines

necessary
+   repairs are performed so that subsequent repairs will not fail
the resource
+   reservation step due to wildly incorrect summary counters. +   Unsuccesful repairs are requeued as long as forward progress on
repairs is
+   made somewhere in the filesystem.
+   Free space in the filesystem is trimmed at the end of phase 4

Quoted 8 lines

if
the
+   filesystem is clean.
+
+5. By the start of this phase, all primary and secondary filesystem
metadata
+   must be correct.

Quoted 5 lines

I think maybe the definitions of primary and secondary metadata should
move up before the phases section.  Otherwise the reader has to skip
ahead to know what that means.

Yep, now primary, secondary, and summary metadata are defined in section
1.  Very good comment.

Quoted 85 lines
Quoted 85 lines

+   Summary counters such as the free space counts and quota resource
counts
+   are checked and corrected.
+   Directory entry names and extended attribute names are checked
for
+   suspicious entries such as control characters or confusing Unicode sequences
+   appearing in names.
+
+6. If the caller asks for a media scan, read all allocated and written data
+   file extents in the filesystem.
+   The ability to use hardware-assisted data file integrity checking
is new
+   to online fsck; neither of the previous tools have this capability.
+   If media errors occur, they will be mapped to the owning files
and reported.
+
+7. Re-check the summary counters and presents the caller with a summary of
+   space usage and file counts.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Steps for Each Scrub Item
+-------------------------
+
+The kernel scrub code uses a three-step strategy for checking

and
repairing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the one aspect of a metadata object represented by a scrub item:
+
+1. The scrub item of intere

st is checked for corruptions; opportunities for
+   optimization; and for values that are directly controlled by the
system
+   administrator but look suspicious.
+   If the item is not corrupt or does not need optimization, resource are
+   released and the positive scan results are returned to userspace.
+   If the item is corrupt or could be optimized but the caller does
not permit
+   this, resources are released and the negative scan results are
returned to
+   userspace.
+   Otherwise, the kernel moves on to the second step. +
+2. The repair function is called to rebuild the data structure. +   Repair functions generally choose rebuild a structure from other
metadata
+   rather than try to salvage the existing structure. +   If the repair fails, the scan results from the first step are returned to
+   userspace.
+   Otherwise, the kernel moves on to the third step.
+
+3. In the third step, the kernel runs the same checks over the new
metadata
+   item to assess the efficacy of the repairs.
+   The results of the reassessment are returned to userspace.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Classification of Metadata
+--------------------------
+
+Each type of metadata object (and therefore each type of scrub

item)
is

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+classified as follows:
+
+Primary Metadata
+````````````````
+
+Metadata structures in this category should be most familiar to

filesystem
+users either because they are directly created by the user or they

Quoted 8 lines
Quoted 2 lines

index
+objects created by the user

I think I would just jump straight into a brief list.  The above is a
bit vague, and documentation that tells you you should already know what it is, doesnt add much.  Again, I think too much poetry might be
why you're having a hard time getting responses.

Done:

- Free space and reference count information

- Inode records and indexes

- Storage mapping information for file data

- Directories

- Extended attributes

- Symbolic links

- Quota limits

- Link counts

Quoted 11 lines
Quoted 1 line

+Most filesystem objects fall into this class.

Most filesystem objects created by users fall into this class, such as
inode, directories, allocation groups and so on.

Quoted 4 lines

+Resource and lock acquisition for scrub code follows the same order
as regular
+filesystem accesses.

Lock acquisition for these resources will follow the same order for scrub as a regular filesystem access.

Quoted 5 lines

Yes, that is clearer.  I think I'll phrase this more actively:

"Scrub obeys the same rules as regular filesystem accesses for resource
and lock acquisition."

Ok, I think that sounds fine

Quoted 70 lines
Quoted 61 lines
Quoted 61 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or

an
inode) that
+owns the item being scrubbed is locked to guard against concurrent
updates.
+The check function examines every record associated with the type
for obvious
+errors and cross-references healthy records against other metadata
to look for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistencies.
+Repairs for this class of scrub item are simple, since the

repair
function
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all
the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and

commits it

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Because ``xfs_scrub`` locks a primary object for the duration of

the
repair,
+this is effectively an offline repair operation performed on a subset of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem.
+This minimizes the complexity of the repair code because it is

not
necessary to
+handle concurrent updates from other threads, nor is it necessary to
access

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and

programs
+trying to access the damaged structure will be blocked until repairs
complete.
+The only infrastructure needed by the repair code are the staging
area for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair

holds is
clear:
+targeted work on individual shards of the filesystem avoids total
loss of

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+service.
+
Quoted 8 lines
Quoted 5 lines

+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,

Hmm, this article is not displaying for me.  If the link is abandoned,
probably there's not much need to keep it around

Quoted 7 lines

The actual paper is not directly available through that ACM link, but the DOI is what I used to track down a paper copy(!) of that paper as published in a journal.

(In turn, that journal is "Advances in Database Technology - EDBT 1992";
I found it in the NYU library.  Amazingly, they sold it to me.)

Oh I see. Dave had replied in a separate thread with a pdf version. That might be a better link so that people do not have to buy a paper copy.

Quoted 43 lines
Quoted 26 lines
Quoted 21 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+*Extending Database Technology*, pp. 293-309, 1992.
+
+Most primary metadata repair functions stage their intermediate

results in an
+in-memory array prior to formatting the new ondisk structure, which
is very
+similar to the list-based algorithm discussed in section 2.3 ("List-
Based

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Algorithms") of Srinivasan.
+However, any data structure builder that maintains a resource

lock
for the

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+duration of the repair is *always* an offline algorithm.
+
+Secondary Metadata
+``````````````````
+
+Metadata structures in this category reflect records found in
Quoted 1 line

primary metadata,

such as rmap and parent pointer attributes.  But they are only needed...

Quoted 1 line

?

Euugh, this section needs some restructuring to get rid of redundant sentences.  How about:

"Metadata structures in this category reflect records found in primary
metadata, but are only needed for online fsck or for reorganization of
the filesystem.

"Secondary metadata include:

- Reverse mapping information

- Directory parent pointers

Quoted 7 lines

"This class of metadata is difficult for scrub to process because scrub
attaches to the secondary object but needs to check primary metadata, which runs counter to the usual order of resource acquisition. Frequently, this means that full filesystems scans are necessary to rebuild the metadata.
Check functions..."

Yes I think that's much clearer :-)

Quoted 76 lines
Quoted 15 lines
Quoted 10 lines

+but are only needed for online fsck or for reorganization of the filesystem.
+Resource and lock acquisition for scrub code do not follow the same
order as
+regular filesystem accesses, and may involve full filesystem scans.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Secondary metadata objects are difficult for scrub to process,
Quoted 4 lines

because scrub
+attaches to the secondary object but needs to check primary metadata, which
+runs counter to the usual order of resource acquisition.

bummer :-(

Yup.

Quoted 19 lines
Quoted 17 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Check functions can be limited in scope to reduce runtime.
+Repairs, however, require a full scan of primary metadata, which

can
take a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+long time to complete.
+Under these conditions, ``xfs_scrub`` cannot lock resources for

the
entire

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+duration of the repair.
+
+Instead, repair functions set up an in-memory staging structure

to
store

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations.
+Depending on the requirements of the specific repair function,

the
staging

Quoted 6 lines
Quoted 4 lines

+index can have the same format as the ondisk structure, or it can
have a design
+specific to that repair function.

...will have either the same format as the ondisk structure or a structure specific to the repair function.

Fixed.

Quoted 16 lines
Quoted 7 lines

+The next step is to release all locks and start the filesystem scan.
+When the repair scanner needs to record an observation, the staging
data are

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+locked long enough to apply the update.
+Simultaneously, the repair function hooks relevant parts of the
Quoted 5 lines

filesystem to
+apply updates to the staging data if the the update pertains to an
object that
+has already been scanned by the index builder.

While a scan is in progress, function hooks are used to apply filesystem updates to both the object and the staging data if the object has already been scanned.

Quoted 1 line

?

The hooks are used to apply updates to the repair staging data, but they
don't apply regular filesystem updates.

The usual process runs something like this:

  Lock -> update -> update -> commit

With a scan in progress, say we hook the second update.  The instruction
flow becomes:

Quoted 8 lines

  Lock -> update -> update -> hook -> update staging data -> commit

Maybe something along the following would be better?

"While the filesystem scan is in progress, the repair function hooks the
filesystem so that it can apply pending filesystem updates to the staging information."

Ok, that sounds clearer then

Quoted 116 lines
Quoted 101 lines
Quoted 46 lines

+Once the scan is done, the owning object is re-locked, the live data
is used to
+write a new ondisk structure, and the repairs are committed atomically.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The hooks are disabled and the staging staging area is freed.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Introducing concurrency helps online repair avoid various

locking
problems, but

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+comes at a high cost to code complexity.
+Live filesystem code has to be hooked so that the repair

function
can observe

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates in progress.
+The staging area has to become a fully functional parallel

structure
so that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates can be merged from the hooks.
+Finally, the hook, the filesystem scan, and the inode locking

model
must be
+sufficiently well integrated that a hook event can decide if a given
update

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+should be applied to the staging structure.
+
+In theory, the scrub implementation could apply these same

techniques for
+primary metadata, but doing so would make it massively more complex
and less

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+performant.
+Programs attempting to access the damaged structures are not

blocked
from
+operation, which may cause application failure or an unplanned filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+shutdown.
+
+Inspiration for the secondary metadata repair strategy was drawn

from section
+2.4 of Srinivasan above, and sections 2 ("NSF: Inded Build

Quoted 8 lines

Without
Side-File")
+and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms
for
+Creating Indexes for Very Large Tables Without Quiescing Updates" +<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.

This one works

Quoted 45 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The sidecar index mentioned above bears some resemblance to the

side
file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+method mentioned in Srinivasan and Mohan.
+Their method consists of an index builder that extracts relevant

record data to
+build the new structure as quickly as possible; and an auxiliary structure that
+captures all updates that would be committed to the index by other
threads were

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the new index already online.
+After the index building scan finishes, the updates recorded in

the
side file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+are applied to the new index.
+To avoid conflicts between the index builder and other writer

threads, the
+builder maintains a publicly visible cursor that tracks the progress
of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scan through the record space.
+To avoid duplication of work between the side file and the index

builder, side
+file updates are elided when the record ID for the update is greater
than the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+cursor position within the record ID space.
+
+To minimize changes to the rest of the codebase, XFS online

repair
keeps the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+replacement index hidden until it's completely ready to go.
+In other words, there is no attempt to expose the keyspace of

the
new index

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+while repair is running.
+The complexity of such an approach would be very high and

perhaps
more

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate to building *new* indices.
+
+**Question**: Can the full scan and live update code used to

facilitate a

Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 467

Quoted 3 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair also be used to implement a comprehensive check?
+
+*Answer*: Probably, though this has not been yet been studied.

I kinda feel like discussion Q&As need to be wrapped up before we can
call things done.  If this is all there was to the answer, then lets
clean out the discussion notes.

Oh, the situation here is worse than that -- in theory, check would be
much stronger if each scrub function employed these live scans to build
a shadow copy of the metadata and then compared the records of both.

Quoted 15 lines

However, that increases the amount of work each scrubber has to do much
higher, and the runtime of those scrubbers would go up.  The other issue
is that live scan hooks would have to proliferate through much more of
the filesystem.  That's rather more invasive to the codebase than most
of fsck, so I want people to look at the usage models for the handful of
scrubbers that really require it before I spread it around elsewhere. Making that kind of change isn't that difficult, but I want to merge this stuff before moving on to experimenting with improvements of that
scale.

I see, well maybe it would be appropriate it to just call it a possible future improvement for now, depending on how the uses cases go and if the demand for it arises.

Quoted 24 lines
Quoted 10 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 471

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Summary Information
+```````````````````
+

Oh, perhaps this section could move up with the other metadata definitions.  That way the reader already has an idea of what these terms are referring to before we get into how they are used during the
phases.

Yeah, I think/hope this will be less of a problem now that section 1 defines all three types of metadata.  The start of this section now reads:

"Metadata structures in this last category summarize the contents of primary metadata records.
These are often used to speed up resource usage queries, and are many times smaller than the primary metadata which they represent.

Examples of summary information include:

- Summary counts of free space and inodes

Quoted 7 lines

- File link counts from directories

- Quota resource usage counts

"Check and repair require full filesystem scans, but resource and lock
acquisition follow the same paths as regular filesystem accesses."

Sounds good, I think that will help a lot

Quoted 171 lines
Quoted 41 lines
Quoted 41 lines

+Metadata structures in this last category summarize the contents of
primary

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata records.
+These are often used to speed up resource usage queries, and are

many times

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+smaller than the primary metadata which they represent.
+Check and repair both require full filesystem scans, but

resource
and lock
+acquisition follow the same paths as regular filesystem accesses.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The superblock summary counters have special requirements due to

the
underlying
+implementation of the incore counters, and will be treated separately.
+Check and repair of the other types of summary counters (quota resource counts
+and file link counts) employ the same filesystem scanning and hooking
+techniques as outlined above, but because the underlying data are
sets of
+integer counters, the staging data need not be a fully functional
mirror of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+ondisk structure.
+
+Inspiration for quota and file link count repair strategies were

drawn from
+sections 2.12 ("Online Index Operations") through 2.14 ("Incremental
View
+Maintenace") of G.  Graefe, `"Concurrent Queries and Updates in Summary Views
+and Their Indexes"
+<
http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf

Quoted 4 lines
Quoted 2 lines
Quoted 1 line

`

_, 2011.

I wonder if these citations would do better as foot notes?  Just to kinda keep the body of the document tidy and flowing well.

Yes, if this were a paginated document.

Quoted 114 lines
Quoted 104 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Since quotas are non-negative integer counts of resource usage,

online
+quotacheck can use the incremental view deltas described in section
2.14 to
+track pending changes to the block and inode usage counts in each
transaction,
+and commit those changes to a dquot side file when the transaction
commits.
+Delta tracking is necessary for dquots because the index builder scans inodes,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+whereas the data structure being rebuilt is an index of dquots.
+Link count checking combines the view deltas and commit step

into
one because
+it sets attributes of the objects being scanned instead of writing
them to a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+separate data structure.
+Each online fsck function will be discussed as case studies

later in
this

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+document.
+
+Risk Management
+---------------
+
+During the development of online fsck, several risk factors were

identified
+that may make the feature unsuitable for certain distributors and
users.
+Steps can be taken to mitigate or eliminate those risks, though at a
cost to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+functionality.
+
+- **Decreased performance**: Adding metadata indices to the

filesystem
+  increases the time cost of persisting changes to disk, and the reverse space
+  mapping and directory parent pointers are no exception. +  System administrators who require the maximum performance can disable the
+  reverse mapping features at format time, though this choice dramatically
+  reduces the ability of online fsck to find inconsistencies and repair them.
+
+- **Incorrect repairs**: As with all software, there might be defects in the
+  software that result in incorrect repairs being written to the filesystem.
+  Systematic fuzz testing (detailed in the next section) is employed
by the
+  authors to find bugs early, but it might not catch everything. +  The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
+  and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose
not to
+  accept this risk.
+  The xfsprogs build system has a configure option (``--enable- scrub=no``) that
+  disables building of the ``xfs_scrub`` binary, though this is not
a risk
+  mitigation if the kernel functionality remains enabled. +
+- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
+  repairable.
+  If the keyspaces of several metadata indices overlap in some manner but a
+  coherent narrative cannot be formed from records collected, then
the repair
+  fails.
+  To reduce the chance that a repair will fail with a dirty transaction and
+  render the filesystem unusable, the online repair functions have
been
+  designed to stage and validate all new records before committing
the new
+  structure.
+
+- **Misbehavior**: Online fsck requires many privileges -- raw IO to
block
+  devices, opening files by handle, ignoring Unix discretionary access control,
+  and the ability to perform administrative changes.
+  Running this automatically in the background scares people, so the
systemd
+  background service is configured to run with only the privileges
required.

Quoted 8 lines

+  Obviously, this cannot address certain problems like the kernel
crashing or
+  deadlocking, but it should be sufficient to prevent the scrub process from
+  escaping and reconfiguring the system.
+  The cron job does not have this protection.
+

I think the fuzz part is one I would consider letting go.  All

Quoted 8 lines

features
need to go through a period of stabilizing, and we cant really control
how some people respond to it, so I don't think this part adds much.  I
think the document would do well to be trimmed where it can so as to
stay more focused

Quoted 2 lines

It took me a minute to realize that this comment applies to the text below it.  Right?

Yes, sorry for confusion :-)

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines

+- **Fuzz Kiddiez**: There are many people now who seem to think that
running
+  automated fuzz testing of ondisk artifacts to find mischevious behavior and
+  spraying exploit code onto the public mailing list for instant zero-day
+  disclosure is somehow of some social benefit.

I want to keep this bit because it keeps happening[2].  Some folks (huawei/alibaba?) have started to try to fix the bugs that their robots
find, and kudos to them!

Quoted 6 lines

You might have noticed that Googlers turned their firehose back on and
once again aren't doing anything to fix the problems they find.  How very Googley of them.

[2] https://lwn.net/Articles/904293/

Alrighty then

Quoted 29 lines
Quoted 15 lines
Quoted 15 lines

+  In the view of this author, the benefit is realized only when the
fuzz
+  operators help to **fix** the flaws, but this opinion apparently
is not
+  widely shared among security "researchers".
+  The XFS maintainers' continuing ability to manage these events presents an
+  ongoing risk to the stability of the development process. +  Automated testing should front-load some of the risk while the feature is
+  considered EXPERIMENTAL.
+

Quoted 8 lines
Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Many of these risks are inherent to software programming.
+Despite this, it is hoped that this new functionality will prove

useful in
+reducing unexpected downtime.

Paraphrasing and reorganizing suggestions aside, I think it looks pretty good

Ok, thank you!

--D

Quoted 1 line

Allison

#13 lore raw

Re: [PATCH 02/14] xfs: document the general theory underlying online fsck design

Quoted 259 lines · Allison Henderson
Quoted 104 lines · Darrick J. Wong
Quoted 77 lines · Allison Henderson
Quoted 76 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the second chapter of the online fsck design documentation. This covers the general theory underlying how online fsck works.

---
 .../filesystems/xfs-online-fsck-design.rst         |  366
++++++++++++++++++++

 1 file changed, 366 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 25717ebb5f80..a03a7b9f0250 100644
@@ -197,3 +197,369 @@ metadata to enable targeted checking and

repair
operations while the system
 is running.
 This capability will be coupled to automatic system management so
that
 autonomous self-healing of XFS maximizes service availability.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+2. Theory of Operation
+======================
+
+Because it is necessary for online fsck to lock and scan live

metadata objects,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+online fsck consists of three separate code components.
+The first is the userspace driver program ``xfs_scrub``, which

is
responsible
+for identifying individual metadata items, scheduling work items for
them,
+reacting to the outcomes appropriately, and reporting results to the
system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+administrator.
+The second and third are in the kernel, which implements

functions
to check

Documentation/filesystems/xfs-online-fsck-design.rst +4 -1
+and repair each type of online fsck work item.
+
++---------------------------------------------------------------
---+
+|

**Note**:
|
++---------------------------------------------------------------
---+
+| For brevity, this document shortens the phrase "online fsck work |
+| item" to "scrub
item".                                           |
++---------------------------------------------------------------
---+

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Scrub item types are delineated in a manner consistent with the

Unix
design
+philosophy, which is to say that each item should handle one aspect
of a

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+metadata structure, and handle it well.
+
+Scope
+-----
+
+In principle, online fsck should be able to check and to repair

everything that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the offline fsck program can handle.
+However, the adjective *online* brings with it the limitation
Quoted 8 lines
Quoted 6 lines

that
online fsck
+cannot deal with anything that prevents the filesystem from going on
line, i.e.
+mounting.

Are there really any other operations that do that other than mount?

No.

Quoted 6 lines

I think this reads cleaner:

By definition, online fsck can only check and repair an online filesystem.  It cannot check mounting operations which start from an
offline state.

Now that I think about this some more, this whole sentence doesn't make
sense.  xfs_scrub can *definitely* detect and fix latent errors that would prevent the /next/ mount from succeeding.  It's only the fuzz test
suite that stumbles over this, and only because xfs_db cannot fuzz mounted filesystems.

Quoted 5 lines

"However, online fsck cannot be running 100% of the time, which means that latent errors may creep in after a scrub completes. If these errors cause the next mount to fail, offline fsck is the only
solution."

Sure, that sounds fair

Quoted 21 lines
Quoted 12 lines
Quoted 12 lines

+This limitation means that maintenance of the offline fsck tool will
continue.
+A second limitation of online fsck is that it must follow the same
resource

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+sharing and lock acquisition rules as the regular filesystem.
+This means that scrub cannot take *any* shortcuts to save time,

because doing

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+so could lead to concurrency problems.
+In other words, online fsck will never be able to fix 100% of
Quoted 8 lines
Quoted 2 lines

the
+inconsistencies that offline fsck can repair,

Hmm, what inconsistencies cannot repaired as a result of the "no shortcut" rule?  I'm all for keeping things short and to the point, but
since this section is about scope, I'd give it at least a brief bullet
list

Quoted 8 lines

Hmm.  I can't think of any off the top of my head.  Given the rewording
earlier, I think it's more accurate to say:

"In other words, online fsck is not a complete replacement for offline
fsck, and a complete run of online fsck may take longer than online fsck."

That makes sense

Quoted 112 lines
Quoted 29 lines
Quoted 29 lines

and a complete run of online fsck

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+may take longer.
+However, both of these limitations are acceptable tradeoffs to

satisfy the
+different motivations of online fsck, which are to **minimize system
downtime**

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+and to **increase predictability of operation**.
+
+.. _scrubphases:
+
+Phases of Work
+--------------
+
+The userspace driver program ``xfs_scrub`` splits the work of

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repairing an entire filesystem into seven phases.
+Each phase concentrates on checking specific types of scrub

items
and depends

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+on the success of all previous phases.
+The seven phases are as follows:
+
+1. Collect geometry information about the mounted filesystem and

computer,
+   discover the online fsck capabilities of the kernel, and open the
+   underlying storage devices.

Quoted 8 lines
Quoted 7 lines

+
+2. Check allocation group metadata, all realtime volume metadata,
and all quota
+   files.
+   Each metadata structure is scheduled as a separate scrub item.

Like an intent item?

No, these scrub items are struct scrub_item objects that exist solely within the userspace program code.

Quoted 7 lines
Quoted 7 lines

+   If corruption is found in the inode header or inode btree and ``xfs_scrub``
+   is permitted to perform repairs, then those scrub items are repaired to
+   prepare for phase 3.
+   Repairs are implemented by resubmitting the scrub item to the kernel with

Quoted 6 lines

If I'm understanding this correctly:
Repairs are implemented as intent items that are queued and committed
just as any filesystem operation.

?

I don't want to go too deep into this prematurely, but...

xfs_scrub (the userspace program) needs to track which metadata objects
have been checked and which ones need repairs.  The current codebase (ab)uses struct xfs_scrub_metadata, but it's very memory inefficient. I replaced it with a new struct scrub_item that stores (a) all the handle information to identify the inode/AG/rt group/whatever; and
(b)
the state of all the checks that can be applied to that item:

struct scrub_item {
        /*
         * Information we need to call the scrub and repair ioctls.
         * Per-AG items should set the ino/gen fields to -1; per- inode
         * items should set sri_agno to -1; and per-fs items should set
         * all three fields to -1.  Or use the macros below. */
        __u64                   sri_ino;
        __u32                   sri_gen;
        __u32                   sri_agno;

        /* Bitmask of scrub types that were scheduled here. */ __u32                   sri_selected;

        /* Scrub item state flags, one for each XFS_SCRUB_TYPE. */ __u8                    sri_state[XFS_SCRUB_TYPE_NR];

        /* Track scrub and repair call retries for each scrub type. */
        __u8                    sri_tries[XFS_SCRUB_TYPE_NR];

        /* Were there any corruption repairs needed? */ bool                    sri_inconsistent:1;

        /* Are we revalidating after repairs? */
        bool                    sri_revalidate:1;
};

The first three fields are passed to the kernel via scrub ioctl and describe a particular xfs domain (files, AGs, etc).  The rest of the structure store state for each type of repair that can be performed against that domain.

IOWs, xfs_scrub uses struct scrub_item objects to generate ioctl calls
to the kernel to check and repair things.  The kernel reads the ioctl information, figures out what needs to be done, and then does the usual
get transaction -> lock things -> make updates -> commit dance to make
corrections to the fs.  Those corrections include log intent items, but
there's no tight coupling between log intent items and scrub_items.

Quoted 4 lines

Side note: The kernel repair code used to use intents to rebuild a structure, but nowadays it use the btree bulk loader code to replace btrees wholesale and in a single atomic commit.  Now we use them primariliy to free preallocated space if the repair fails.

Quoted 6 lines

Oh ok, well how about just:

"Repairs are implemented by resubmitting the scrub item to the kernel through a designated ioctl with..."

?

How about:

"Repairs are implemented by using the information in the scrub item to resubmit the kernel scrub call with the repair flag enabled; this is discussed in the next section. Optimizations and all other repairs are deferred to phase 4."

?

Quoted 278 lines
Quoted 34 lines
Quoted 6 lines
Quoted 4 lines

+   the repair flag enabled; this is discussed in the next section.
+   Optimizations and all other repairs are deferred to phase 4.

I guess I'll come back to it.

Quoted 7 lines
Quoted 6 lines

+
+3. Check all metadata of every file in the filesystem. +   Each metadata structure is also scheduled as a separate scrub item.
+   If repairs are needed, ``xfs_scrub`` is permitted to perform repairs,

If repairs are needed and ``xfs_scrub`` is permitted

Fixed.

Quoted 14 lines

?

Quoted 5 lines

+   and there were no problems detected during phase 2, then those
scrub items
+   are repaired.
+   Optimizations and unsuccessful repairs are deferred to phase

Quoted 8 lines

4.
+
+4. All remaining repairs and scheduled optimizations are performed
during this
+   phase, if the caller permits them.
+   Before starting repairs, the summary counters are checked and any

Quoted 3 lines

Did we talk about summary counters yet?  Maybe worth a blub. Otherwise
this may not make sense with out skipping ahead or into the code

Quoted 7 lines

Nope.  I'll add that to the previous patch when I introduce primary and
secondary metadata.  Good catch!

"Summary metadata, as the name implies, condense information contained
in primary metadata for performance reasons."

Ok, sounds good then

Quoted 154 lines
Quoted 19 lines
Quoted 11 lines

necessary
+   repairs are performed so that subsequent repairs will not fail
the resource
+   reservation step due to wildly incorrect summary counters. +   Unsuccesful repairs are requeued as long as forward progress on
repairs is
+   made somewhere in the filesystem.
+   Free space in the filesystem is trimmed at the end of phase 4

Quoted 8 lines

if
the
+   filesystem is clean.
+
+5. By the start of this phase, all primary and secondary filesystem
metadata
+   must be correct.

Quoted 5 lines

I think maybe the definitions of primary and secondary metadata should
move up before the phases section.  Otherwise the reader has to skip
ahead to know what that means.

Yep, now primary, secondary, and summary metadata are defined in section
1.  Very good comment.

Quoted 85 lines
Quoted 85 lines

+   Summary counters such as the free space counts and quota resource
counts
+   are checked and corrected.
+   Directory entry names and extended attribute names are checked
for
+   suspicious entries such as control characters or confusing Unicode sequences
+   appearing in names.
+
+6. If the caller asks for a media scan, read all allocated and written data
+   file extents in the filesystem.
+   The ability to use hardware-assisted data file integrity checking
is new
+   to online fsck; neither of the previous tools have this capability.
+   If media errors occur, they will be mapped to the owning files
and reported.
+
+7. Re-check the summary counters and presents the caller with a summary of
+   space usage and file counts.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Steps for Each Scrub Item
+-------------------------
+
+The kernel scrub code uses a three-step strategy for checking

and
repairing

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the one aspect of a metadata object represented by a scrub item:
+
+1. The scrub item of intere

st is checked for corruptions; opportunities for
+   optimization; and for values that are directly controlled by the
system
+   administrator but look suspicious.
+   If the item is not corrupt or does not need optimization, resource are
+   released and the positive scan results are returned to userspace.
+   If the item is corrupt or could be optimized but the caller does
not permit
+   this, resources are released and the negative scan results are
returned to
+   userspace.
+   Otherwise, the kernel moves on to the second step. +
+2. The repair function is called to rebuild the data structure. +   Repair functions generally choose rebuild a structure from other
metadata
+   rather than try to salvage the existing structure. +   If the repair fails, the scan results from the first step are returned to
+   userspace.
+   Otherwise, the kernel moves on to the third step.
+
+3. In the third step, the kernel runs the same checks over the new
metadata
+   item to assess the efficacy of the repairs.
+   The results of the reassessment are returned to userspace.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Classification of Metadata
+--------------------------
+
+Each type of metadata object (and therefore each type of scrub

item)
is

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+classified as follows:
+
+Primary Metadata
+````````````````
+
+Metadata structures in this category should be most familiar to

filesystem
+users either because they are directly created by the user or they

Quoted 8 lines
Quoted 2 lines

index
+objects created by the user

I think I would just jump straight into a brief list.  The above is a
bit vague, and documentation that tells you you should already know what it is, doesnt add much.  Again, I think too much poetry might be
why you're having a hard time getting responses.

Done:

- Free space and reference count information

- Inode records and indexes

- Storage mapping information for file data

- Directories

- Extended attributes

- Symbolic links

- Quota limits

- Link counts

Quoted 11 lines
Quoted 1 line

+Most filesystem objects fall into this class.

Most filesystem objects created by users fall into this class, such as
inode, directories, allocation groups and so on.

Quoted 4 lines

+Resource and lock acquisition for scrub code follows the same order
as regular
+filesystem accesses.

Lock acquisition for these resources will follow the same order for scrub as a regular filesystem access.

Quoted 5 lines

Yes, that is clearer.  I think I'll phrase this more actively:

"Scrub obeys the same rules as regular filesystem accesses for resource
and lock acquisition."

Ok, I think that sounds fine

Quoted 74 lines
Quoted 61 lines
Quoted 61 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Primary metadata objects are the simplest for scrub to process.
+The principal filesystem object (either an allocation group or

an
inode) that
+owns the item being scrubbed is locked to guard against concurrent
updates.
+The check function examines every record associated with the type
for obvious
+errors and cross-references healthy records against other metadata
to look for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistencies.
+Repairs for this class of scrub item are simple, since the

repair
function
+starts by holding all the resources acquired in the previous step.
+The repair function scans available metadata as needed to record all
the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations needed to complete the structure.
+Next, it stages the observations in a new ondisk structure and

commits it

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+atomically to complete the repair.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Because ``xfs_scrub`` locks a primary object for the duration of

the
repair,
+this is effectively an offline repair operation performed on a subset of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem.
+This minimizes the complexity of the repair code because it is

not
necessary to
+handle concurrent updates from other threads, nor is it necessary to
access

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+any other part of the filesystem.
+As a result, indexed structures can be rebuilt very quickly, and

programs
+trying to access the damaged structure will be blocked until repairs
complete.
+The only infrastructure needed by the repair code are the staging
area for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations and a means to write new structures to disk.
+Despite these limitations, the advantage that online repair

holds is
clear:
+targeted work on individual shards of the filesystem avoids total
loss of

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+service.
+
Quoted 8 lines
Quoted 5 lines

+This mechanism is described in section 2.1 ("Off-Line Algorithm") of
+V. Srinivasan and M. J. Carey, `"Performance of On-Line Index Construction
+Algorithms" <https://dl.acm.org/doi/10.5555/645336.649870>`_,

Hmm, this article is not displaying for me.  If the link is abandoned,
probably there's not much need to keep it around

The actual paper is not directly available through that ACM link, but the DOI is what I used to track down a paper copy(!) of that paper as published in a journal.

Quoted 6 lines
Quoted 3 lines

(In turn, that journal is "Advances in Database Technology - EDBT 1992";
I found it in the NYU library.  Amazingly, they sold it to me.)

Oh I see. Dave had replied in a separate thread with a pdf version. That might be a better link so that people do not have to buy a paper copy.

Yep, updated, thanks all!

Quoted 271 lines
Quoted 43 lines
Quoted 26 lines
Quoted 21 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+*Extending Database Technology*, pp. 293-309, 1992.
+
+Most primary metadata repair functions stage their intermediate

results in an
+in-memory array prior to formatting the new ondisk structure, which
is very
+similar to the list-based algorithm discussed in section 2.3 ("List-
Based

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Algorithms") of Srinivasan.
+However, any data structure builder that maintains a resource

lock
for the

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+duration of the repair is *always* an offline algorithm.
+
+Secondary Metadata
+``````````````````
+
+Metadata structures in this category reflect records found in
Quoted 1 line

primary metadata,

such as rmap and parent pointer attributes.  But they are only needed...

Quoted 1 line

?

Euugh, this section needs some restructuring to get rid of redundant sentences.  How about:

"Metadata structures in this category reflect records found in primary
metadata, but are only needed for online fsck or for reorganization of
the filesystem.

"Secondary metadata include:

- Reverse mapping information

- Directory parent pointers

Quoted 7 lines

"This class of metadata is difficult for scrub to process because scrub
attaches to the secondary object but needs to check primary metadata, which runs counter to the usual order of resource acquisition. Frequently, this means that full filesystems scans are necessary to rebuild the metadata.
Check functions..."

Yes I think that's much clearer :-)

Quoted 76 lines
Quoted 15 lines
Quoted 10 lines

+but are only needed for online fsck or for reorganization of the filesystem.
+Resource and lock acquisition for scrub code do not follow the same
order as
+regular filesystem accesses, and may involve full filesystem scans.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Secondary metadata objects are difficult for scrub to process,
Quoted 4 lines

because scrub
+attaches to the secondary object but needs to check primary metadata, which
+runs counter to the usual order of resource acquisition.

bummer :-(

Yup.

Quoted 19 lines
Quoted 17 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Check functions can be limited in scope to reduce runtime.
+Repairs, however, require a full scan of primary metadata, which

can
take a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+long time to complete.
+Under these conditions, ``xfs_scrub`` cannot lock resources for

the
entire

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+duration of the repair.
+
+Instead, repair functions set up an in-memory staging structure

to
store

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+observations.
+Depending on the requirements of the specific repair function,

the
staging

Quoted 6 lines
Quoted 4 lines

+index can have the same format as the ondisk structure, or it can
have a design
+specific to that repair function.

...will have either the same format as the ondisk structure or a structure specific to the repair function.

Fixed.

Quoted 16 lines
Quoted 7 lines

+The next step is to release all locks and start the filesystem scan.
+When the repair scanner needs to record an observation, the staging
data are

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+locked long enough to apply the update.
+Simultaneously, the repair function hooks relevant parts of the
Quoted 5 lines

filesystem to
+apply updates to the staging data if the the update pertains to an
object that
+has already been scanned by the index builder.

While a scan is in progress, function hooks are used to apply filesystem updates to both the object and the staging data if the object has already been scanned.

Quoted 1 line

?

The hooks are used to apply updates to the repair staging data, but they
don't apply regular filesystem updates.

The usual process runs something like this:

  Lock -> update -> update -> commit

With a scan in progress, say we hook the second update.  The instruction
flow becomes:

Quoted 8 lines

  Lock -> update -> update -> hook -> update staging data -> commit

Maybe something along the following would be better?

"While the filesystem scan is in progress, the repair function hooks the
filesystem so that it can apply pending filesystem updates to the staging information."

Ok, that sounds clearer then

Quoted 116 lines
Quoted 101 lines
Quoted 46 lines

+Once the scan is done, the owning object is re-locked, the live data
is used to
+write a new ondisk structure, and the repairs are committed atomically.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+The hooks are disabled and the staging staging area is freed.
+Finally, the storage from the old data structure are carefully

reaped.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Introducing concurrency helps online repair avoid various

locking
problems, but

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+comes at a high cost to code complexity.
+Live filesystem code has to be hooked so that the repair

function
can observe

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates in progress.
+The staging area has to become a fully functional parallel

structure
so that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+updates can be merged from the hooks.
+Finally, the hook, the filesystem scan, and the inode locking

model
must be
+sufficiently well integrated that a hook event can decide if a given
update

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+should be applied to the staging structure.
+
+In theory, the scrub implementation could apply these same

techniques for
+primary metadata, but doing so would make it massively more complex
and less

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+performant.
+Programs attempting to access the damaged structures are not

blocked
from
+operation, which may cause application failure or an unplanned filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+shutdown.
+
+Inspiration for the secondary metadata repair strategy was drawn

from section
+2.4 of Srinivasan above, and sections 2 ("NSF: Inded Build

Quoted 8 lines

Without
Side-File")
+and 3.1.1 ("Duplicate Key Insert Problem") in C. Mohan, `"Algorithms
for
+Creating Indexes for Very Large Tables Without Quiescing Updates" +<https://dl.acm.org/doi/10.1145/130283.130337>`_, 1992.

This one works

Quoted 45 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The sidecar index mentioned above bears some resemblance to the

side
file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+method mentioned in Srinivasan and Mohan.
+Their method consists of an index builder that extracts relevant

record data to
+build the new structure as quickly as possible; and an auxiliary structure that
+captures all updates that would be committed to the index by other
threads were

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the new index already online.
+After the index building scan finishes, the updates recorded in

the
side file

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+are applied to the new index.
+To avoid conflicts between the index builder and other writer

threads, the
+builder maintains a publicly visible cursor that tracks the progress
of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scan through the record space.
+To avoid duplication of work between the side file and the index

builder, side
+file updates are elided when the record ID for the update is greater
than the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+cursor position within the record ID space.
+
+To minimize changes to the rest of the codebase, XFS online

repair
keeps the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+replacement index hidden until it's completely ready to go.
+In other words, there is no attempt to expose the keyspace of

the
new index

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+while repair is running.
+The complexity of such an approach would be very high and

perhaps
more

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate to building *new* indices.
+
+**Question**: Can the full scan and live update code used to

facilitate a

Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 467

Quoted 3 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair also be used to implement a comprehensive check?
+
+*Answer*: Probably, though this has not been yet been studied.

I kinda feel like discussion Q&As need to be wrapped up before we can
call things done.  If this is all there was to the answer, then lets
clean out the discussion notes.

Oh, the situation here is worse than that -- in theory, check would be
much stronger if each scrub function employed these live scans to build
a shadow copy of the metadata and then compared the records of both.

Quoted 15 lines

However, that increases the amount of work each scrubber has to do much
higher, and the runtime of those scrubbers would go up.  The other issue
is that live scan hooks would have to proliferate through much more of
the filesystem.  That's rather more invasive to the codebase than most
of fsck, so I want people to look at the usage models for the handful of
scrubbers that really require it before I spread it around elsewhere. Making that kind of change isn't that difficult, but I want to merge this stuff before moving on to experimenting with improvements of that
scale.

Quoted 3 lines

I see, well maybe it would be appropriate it to just call it a possible future improvement for now, depending on how the uses cases go and if the demand for it arises.

I'll go relabel these as "Future Work Questions". Thanks for continuing through! :)

--D

Quoted 261 lines
Quoted 24 lines
Quoted 10 lines

Documentation/filesystems/xfs-online-fsck-design.rst metadata to enable targeted checking and repair operations while the system · line 471

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Summary Information
+```````````````````
+

Oh, perhaps this section could move up with the other metadata definitions.  That way the reader already has an idea of what these terms are referring to before we get into how they are used during the
phases.

Yeah, I think/hope this will be less of a problem now that section 1 defines all three types of metadata.  The start of this section now reads:

"Metadata structures in this last category summarize the contents of primary metadata records.
These are often used to speed up resource usage queries, and are many times smaller than the primary metadata which they represent.

Examples of summary information include:

- Summary counts of free space and inodes

Quoted 7 lines

- File link counts from directories

- Quota resource usage counts

"Check and repair require full filesystem scans, but resource and lock
acquisition follow the same paths as regular filesystem accesses."

Sounds good, I think that will help a lot

Quoted 171 lines
Quoted 41 lines
Quoted 41 lines

+Metadata structures in this last category summarize the contents of
primary

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata records.
+These are often used to speed up resource usage queries, and are

many times

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+smaller than the primary metadata which they represent.
+Check and repair both require full filesystem scans, but

resource
and lock
+acquisition follow the same paths as regular filesystem accesses.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The superblock summary counters have special requirements due to

the
underlying
+implementation of the incore counters, and will be treated separately.
+Check and repair of the other types of summary counters (quota resource counts
+and file link counts) employ the same filesystem scanning and hooking
+techniques as outlined above, but because the underlying data are
sets of
+integer counters, the staging data need not be a fully functional
mirror of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+ondisk structure.
+
+Inspiration for quota and file link count repair strategies were

drawn from
+sections 2.12 ("Online Index Operations") through 2.14 ("Incremental
View
+Maintenace") of G.  Graefe, `"Concurrent Queries and Updates in Summary Views
+and Their Indexes"
+<
http://www.odbms.org/wp-content/uploads/2014/06/Increment-locks.pdf

Quoted 4 lines
Quoted 2 lines
Quoted 1 line

`

_, 2011.

I wonder if these citations would do better as foot notes?  Just to kinda keep the body of the document tidy and flowing well.

Yes, if this were a paginated document.

Quoted 114 lines
Quoted 104 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Since quotas are non-negative integer counts of resource usage,

online
+quotacheck can use the incremental view deltas described in section
2.14 to
+track pending changes to the block and inode usage counts in each
transaction,
+and commit those changes to a dquot side file when the transaction
commits.
+Delta tracking is necessary for dquots because the index builder scans inodes,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+whereas the data structure being rebuilt is an index of dquots.
+Link count checking combines the view deltas and commit step

into
one because
+it sets attributes of the objects being scanned instead of writing
them to a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+separate data structure.
+Each online fsck function will be discussed as case studies

later in
this

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+document.
+
+Risk Management
+---------------
+
+During the development of online fsck, several risk factors were

identified
+that may make the feature unsuitable for certain distributors and
users.
+Steps can be taken to mitigate or eliminate those risks, though at a
cost to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+functionality.
+
+- **Decreased performance**: Adding metadata indices to the

filesystem
+  increases the time cost of persisting changes to disk, and the reverse space
+  mapping and directory parent pointers are no exception. +  System administrators who require the maximum performance can disable the
+  reverse mapping features at format time, though this choice dramatically
+  reduces the ability of online fsck to find inconsistencies and repair them.
+
+- **Incorrect repairs**: As with all software, there might be defects in the
+  software that result in incorrect repairs being written to the filesystem.
+  Systematic fuzz testing (detailed in the next section) is employed
by the
+  authors to find bugs early, but it might not catch everything. +  The kernel build system provides Kconfig options (``CONFIG_XFS_ONLINE_SCRUB``
+  and ``CONFIG_XFS_ONLINE_REPAIR``) to enable distributors to choose
not to
+  accept this risk.
+  The xfsprogs build system has a configure option (``--enable- scrub=no``) that
+  disables building of the ``xfs_scrub`` binary, though this is not
a risk
+  mitigation if the kernel functionality remains enabled. +
+- **Inability to repair**: Sometimes, a filesystem is too badly damaged to be
+  repairable.
+  If the keyspaces of several metadata indices overlap in some manner but a
+  coherent narrative cannot be formed from records collected, then
the repair
+  fails.
+  To reduce the chance that a repair will fail with a dirty transaction and
+  render the filesystem unusable, the online repair functions have
been
+  designed to stage and validate all new records before committing
the new
+  structure.
+
+- **Misbehavior**: Online fsck requires many privileges -- raw IO to
block
+  devices, opening files by handle, ignoring Unix discretionary access control,
+  and the ability to perform administrative changes.
+  Running this automatically in the background scares people, so the
systemd
+  background service is configured to run with only the privileges
required.

Quoted 8 lines

+  Obviously, this cannot address certain problems like the kernel
crashing or
+  deadlocking, but it should be sufficient to prevent the scrub process from
+  escaping and reconfiguring the system.
+  The cron job does not have this protection.
+

I think the fuzz part is one I would consider letting go.  All

Quoted 8 lines

features
need to go through a period of stabilizing, and we cant really control
how some people respond to it, so I don't think this part adds much.  I
think the document would do well to be trimmed where it can so as to
stay more focused

Quoted 2 lines

It took me a minute to realize that this comment applies to the text below it.  Right?

Yes, sorry for confusion :-)

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines

+- **Fuzz Kiddiez**: There are many people now who seem to think that
running
+  automated fuzz testing of ondisk artifacts to find mischevious behavior and
+  spraying exploit code onto the public mailing list for instant zero-day
+  disclosure is somehow of some social benefit.

I want to keep this bit because it keeps happening[2].  Some folks (huawei/alibaba?) have started to try to fix the bugs that their robots
find, and kudos to them!

Quoted 6 lines

You might have noticed that Googlers turned their firehose back on and
once again aren't doing anything to fix the problems they find.  How very Googley of them.

[2] https://lwn.net/Articles/904293/

Alrighty then

Quoted 29 lines
Quoted 15 lines
Quoted 15 lines

+  In the view of this author, the benefit is realized only when the
fuzz
+  operators help to **fix** the flaws, but this opinion apparently
is not
+  widely shared among security "researchers".
+  The XFS maintainers' continuing ability to manage these events presents an
+  ongoing risk to the stability of the development process. +  Automated testing should front-load some of the risk while the feature is
+  considered EXPERIMENTAL.
+

Quoted 8 lines
Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Many of these risks are inherent to software programming.
+Despite this, it is hoped that this new functionality will prove

useful in
+reducing unexpected downtime.

Paraphrasing and reorganizing suggestions aside, I think it looks pretty good

Ok, thank you!

--D

Quoted 1 line

Allison

#14 lore raw

PATCH 3/14 xfs: document the testing plan for online fsck

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the third chapter of the online fsck design documentation. This covers the testing plan to make sure that both online and offline fsck can detect arbitrary problems and correct them without making things worse.

---
 .../filesystems/xfs-online-fsck-design.rst         |  187 ++++++++++++++++++++
 1 file changed, 187 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +187 -0
index a03a7b9f0250..d630b6bdbe4a 100644
@@ -563,3 +563,190 @@ functionality.
 Many of these risks are inherent to software programming.
 Despite this, it is hoped that this new functionality will prove useful in
 reducing unexpected downtime.
+
+3. Testing Plan
+===============
+
+As stated before, fsck tools have three main goals:
+
+1. Detect inconsistencies in the metadata;
+
+2. Eliminate those inconsistencies; and
+
+3. Minimize further loss of data.
+
+Demonstrations of correct operation are necessary to build users' confidence
+that the software behaves within expectations.
+Unfortunately, it was not really feasible to perform regular exhaustive testing
+of every aspect of a fsck tool until the introduction of low-cost virtual
+machines with high-IOPS storage.
+With ample hardware availability in mind, the testing strategy for the online
+fsck project involves differential analysis against the existing fsck tools and
+systematic testing of every attribute of every type of metadata object.
+Testing can be split into four major categories, as discussed below.
+
+Integrated Testing with fstests
+-------------------------------
+
+The primary goal of any free software QA effort is to make testing as
+inexpensive and widespread as possible to maximize the scaling advantages of
+community.
+In other words, testing should maximize the breadth of filesystem configuration
+scenarios and hardware setups.
+This improves code quality by enabling the authors of online fsck to find and
+fix bugs early, and helps developers of new features to find integration
+issues earlier in their development effort.
+
+The Linux filesystem community shares a common QA testing suite,
+`fstests <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/>`_, for
+functional and regression testing.
+Even before development work began on online fsck, fstests (when run on XFS)
+would run both the ``xfs_check`` and ``xfs_repair -n`` commands on the test and
+scratch filesystems between each test.
+This provides a level of assurance that the kernel and the fsck tools stay in
+alignment about what constitutes consistent metadata.
+During development of the online checking code, fstests was modified to run
+``xfs_scrub -n`` between each test to ensure that the new checking code
+produces the same results as the two existing fsck tools.
+
+To start development of online repair, fstests was modified to run
+``xfs_repair`` to rebuild the filesystem's metadata indices between tests.
+This ensures that offline repair does not crash, leave a corrupt filesystem
+after it exists, or trigger complaints from the online check.
+This also established a baseline for what can and cannot be repaired offline.
+To complete the first phase of development of online repair, fstests was
+modified to be able to run ``xfs_scrub`` in a "force rebuild" mode.
+This enables a comparison of the effectiveness of online repair as compared to
+the existing offline repair tools.
+
+General Fuzz Testing of Metadata Blocks
+---------------------------------------
+
+XFS benefits greatly from having a very robust debugging tool, ``xfs_db``.
+
+Before development of online fsck even began, a set of fstests were created
+to test the rather common fault that entire metadata blocks get corrupted.
+This required the creation of fstests library code that can create a filesystem
+containing every possible type of metadata object.
+Next, individual test cases were created to create a test filesystem, identify
+a single block of a specific type of metadata object, trash it with the
+existing ``blocktrash`` command in ``xfs_db``, and test the reaction of a
+particular metadata validation strategy.
+
+This earlier test suite enabled XFS developers to test the ability of the
+in-kernel validation functions and the ability of the offline fsck tool to
+detect and eliminate the inconsistent metadata.
+This part of the test suite was extended to cover online fsck in exactly the
+same manner.
+
+In other words, for a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem:
+
+  * Write garbage to it
+
+  * Test the reactions of:
+
+    1. The kernel verifiers to stop obviously bad metadata
+    2. Offline repair (``xfs_repair``) to detect and fix
+    3. Online repair (``xfs_scrub``) to detect and fix
+
+Targeted Fuzz Testing of Metadata Records
+-----------------------------------------
+
+A quick conversation with the other XFS developers revealed that the existing
+test infrastructure could be extended to provide a much more powerful
+facility: targeted fuzz testing of every metadata field of every metadata
+object in the filesystem.
+``xfs_db`` can modify every field of every metadata structure in every
+block in the filesystem to simulate the effects of memory corruption and
+software bugs.
+Given that fstests already contains the ability to create a filesystem
+containing every metadata format known to the filesystem, ``xfs_db`` can be
+used to perform exhaustive fuzz testing!
+
+For a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem...
+
+  * For each record inside that metadata object...
+
+    * For each field inside that record...
+
+      * For each conceivable type of transformation that can be applied to a bit field...
+
+        1. Clear all bits
+        2. Set all bits
+        3. Toggle the most significant bit
+        4. Toggle the middle bit
+        5. Toggle the least significant bit
+        6. Add a small quantity
+        7. Subtract a small quantity
+        8. Randomize the contents
+
+        * ...test the reactions of:
+
+          1. The kernel verifiers to stop obviously bad metadata
+          2. Offline checking (``xfs_repair -n``)
+          3. Offline repair (``xfs_repair``)
+          4. Online checking (``xfs_scrub -n``)
+          5. Online repair (``xfs_scrub``)
+          6. Both repair tools (``xfs_scrub`` and then ``xfs_repair`` if online repair doesn't succeed)
+
+This is quite the combinatoric explosion!
+
+Fortunately, having this much test coverage makes it easy for XFS developers to
+check the responses of XFS' fsck tools.
+Since the introduction of the fuzz testing framework, these tests have been
+used to discover incorrect repair code and missing functionality for entire
+classes of metadata objects in ``xfs_repair``.
+The enhanced testing was used to finalize the deprecation of ``xfs_check`` by
+confirming that ``xfs_repair`` could detect at least as many corruptions as
+the older tool.
+
+These tests have been very valuable for ``xfs_scrub`` in the same ways -- they
+allow the online fsck developers to compare online fsck against offline fsck,
+and they enable XFS developers to find deficiencies in the code base.
+
+Proposed patchsets include
+`general fuzzer improvements
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements>`_,
+`fuzzing baselines
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline>`_,
+and `improvements in fuzz testing comprehensiveness
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing>`_.
+
+Stress Testing
+--------------
+
+A unique requirement to online fsck is the ability to operate on a filesystem
+concurrently with regular workloads.
+Although it is of course impossible to run ``xfs_scrub`` with *zero* observable
+impact on the running system, the online repair code should never introduce
+inconsistencies into the filesystem metadata, and regular workloads should
+never notice resource starvation.
+To verify that these conditions are being met, fstests has been enhanced in
+the following ways:
+
+* For each scrub item type, create a test to exercise checking that item type
+  while running ``fsstress``.
+* For each scrub item type, create a test to exercise repairing that item type
+  while running ``fsstress``.
+* Race ``fsstress`` and ``xfs_scrub -n`` to ensure that checking the whole
+  filesystem doesn't cause problems.
+* Race ``fsstress`` and ``xfs_scrub`` in force-rebuild mode to ensure that
+  force-repairing the whole filesystem doesn't cause problems.
+* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  freezing and thawing the filesystem.
+* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  remounting the filesystem read-only and read-write.
+* The same, but running ``fsx`` instead of ``fsstress``.  (Not done yet?)
+
+Success is defined by the ability to run all of these tests without observing
+any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
+check warnings, or any other sort of mischief.
+
+Proposed patchsets include `general stress testing
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_
+and the `evolution of existing per-function stress testing
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_.
Reviewed-by on v1 3/14
#15 lore raw

Re: [PATCH 03/14] xfs: document the testing plan for online fsck

Quoted 150 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the third chapter of the online fsck design documentation. This
covers the testing plan to make sure that both online and offline fsck
can detect arbitrary problems and correct them without making things worse.

---
 .../filesystems/xfs-online-fsck-design.rst         |  187
++++++++++++++++++++

 1 file changed, 187 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index a03a7b9f0250..d630b6bdbe4a 100644
@@ -563,3 +563,190 @@ functionality.

 Many of these risks are inherent to software programming. Despite this, it is hoped that this new functionality will prove useful in
 reducing unexpected downtime.

Documentation/filesystems/xfs-online-fsck-design.rst +13 -0
+
+3. Testing Plan
+===============
+
+As stated before, fsck tools have three main goals:
+
+1. Detect inconsistencies in the metadata;
+
+2. Eliminate those inconsistencies; and
+
+3. Minimize further loss of data.
+
+Demonstrations of correct operation are necessary to build users'

confidence

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+that the software behaves within expectations.
+Unfortunately, it was not really feasible to perform regular

exhaustive testing
+of every aspect of a fsck tool until the introduction of low-cost virtual

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+machines with high-IOPS storage.
+With ample hardware availability in mind, the testing strategy for

the online
+fsck project involves differential analysis against the existing fsck tools and
+systematic testing of every attribute of every type of metadata object.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+Testing can be split into four major categories, as discussed below.
+
+Integrated Testing with fstests
+-------------------------------
+
+The primary goal of any free software QA effort is to make testing

as
+inexpensive and widespread as possible to maximize the scaling advantages of

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+community.
+In other words, testing should maximize the breadth of filesystem

configuration

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scenarios and hardware setups.
+This improves code quality by enabling the authors of online fsck to

find and
+fix bugs early, and helps developers of new features to find integration

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+issues earlier in their development effort.
+
+The Linux filesystem community shares a common QA testing suite,
+`fstests

<https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/>`_, for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+functional and regression testing.
+Even before development work began on online fsck, fstests (when run

on XFS)
+would run both the ``xfs_check`` and ``xfs_repair -n`` commands on the test and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scratch filesystems between each test.
+This provides a level of assurance that the kernel and the fsck

tools stay in

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+alignment about what constitutes consistent metadata.
+During development of the online checking code, fstests was modified

to run
+``xfs_scrub -n`` between each test to ensure that the new checking code

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+produces the same results as the two existing fsck tools.
+
+To start development of online repair, fstests was modified to run
+``xfs_repair`` to rebuild the filesystem's metadata indices between

tests.
+This ensures that offline repair does not crash, leave a corrupt filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+after it exists, or trigger complaints from the online check.
+This also established a baseline for what can and cannot be repaired

offline.
+To complete the first phase of development of online repair, fstests was

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+modified to be able to run ``xfs_scrub`` in a "force rebuild" mode.
+This enables a comparison of the effectiveness of online repair as

compared to

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+the existing offline repair tools.
+
+General Fuzz Testing of Metadata Blocks
+---------------------------------------
+
+XFS benefits greatly from having a very robust debugging tool,

``xfs_db``.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Before development of online fsck even began, a set of fstests were

created
+to test the rather common fault that entire metadata blocks get corrupted.
+This required the creation of fstests library code that can create a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+containing every possible type of metadata object.
+Next, individual test cases were created to create a test

filesystem, identify
+a single block of a specific type of metadata object, trash it with the
+existing ``blocktrash`` command in ``xfs_db``, and test the reaction of a

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+particular metadata validation strategy.
+
+This earlier test suite enabled XFS developers to test the ability

of the
+in-kernel validation functions and the ability of the offline fsck tool to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+detect and eliminate the inconsistent metadata.
+This part of the test suite was extended to cover online fsck in

exactly the

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+same manner.
+
+In other words, for a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem:
+
+  * Write garbage to it
+
+  * Test the reactions of:
+
+    1. The kernel verifiers to stop obviously bad metadata
+    2. Offline repair (``xfs_repair``) to detect and fix
+    3. Online repair (``xfs_scrub``) to detect and fix
+
+Targeted Fuzz Testing of Metadata Records
+-----------------------------------------
+
+A quick conversation with the other XFS developers revealed that the
Quoted 2 lines

existing
+test infrastructure could be extended to provide

"The testing plan for ofsck includes extending the existing test infrastructure to provide..."

Took me a moment to notice we're not talking about history any more....

Quoted 43 lines

a much more powerful
+facility: targeted fuzz testing of every metadata field of every metadata
+object in the filesystem.
+``xfs_db`` can modify every field of every metadata structure in every
+block in the filesystem to simulate the effects of memory corruption and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+software bugs.
+Given that fstests already contains the ability to create a

filesystem
+containing every metadata format known to the filesystem, ``xfs_db`` can be

Documentation/filesystems/xfs-online-fsck-design.rst +11 -0
+used to perform exhaustive fuzz testing!
+
+For a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem...
+
+  * For each record inside that metadata object...
+
+    * For each field inside that record...
+
+      * For each conceivable type of transformation that can be

applied to a bit field...

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+
+        1. Clear all bits
+        2. Set all bits
+        3. Toggle the most significant bit
+        4. Toggle the middle bit
+        5. Toggle the least significant bit
+        6. Add a small quantity
+        7. Subtract a small quantity
+        8. Randomize the contents
+
+        * ...test the reactions of:
+
+          1. The kernel verifiers to stop obviously bad metadata
+          2. Offline checking (``xfs_repair -n``)
+          3. Offline repair (``xfs_repair``)
+          4. Online checking (``xfs_scrub -n``)
+          5. Online repair (``xfs_scrub``)
+          6. Both repair tools (``xfs_scrub`` and then
Quoted 1 line

``xfs_repair`` if online repair doesn't succeed)

I like the indented bullet list format tho

Quoted 77 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+This is quite the combinatoric explosion!
+
+Fortunately, having this much test coverage makes it easy for XFS

developers to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+check the responses of XFS' fsck tools.
+Since the introduction of the fuzz testing framework, these tests

have been
+used to discover incorrect repair code and missing functionality for entire

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+classes of metadata objects in ``xfs_repair``.
+The enhanced testing was used to finalize the deprecation of

``xfs_check`` by
+confirming that ``xfs_repair`` could detect at least as many corruptions as

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the older tool.
+
+These tests have been very valuable for ``xfs_scrub`` in the same

ways -- they
+allow the online fsck developers to compare online fsck against offline fsck,
+and they enable XFS developers to find deficiencies in the code base.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets include
+`general fuzzer improvements
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=fuzzer-improvements>`_,
+`fuzzing baselines
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=fuzz-baseline>`_,
+and `improvements in fuzz testing comprehensiveness +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=more-fuzz-testing>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Stress Testing
+--------------
+
+A unique requirement to online fsck is the ability to operate on a

filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+concurrently with regular workloads.
+Although it is of course impossible to run ``xfs_scrub`` with *zero*

observable
+impact on the running system, the online repair code should never introduce
+inconsistencies into the filesystem metadata, and regular workloads should

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+never notice resource starvation.
+To verify that these conditions are being met, fstests has been

enhanced in

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the following ways:
+
+* For each scrub item type, create a test to exercise checking that

item type
+  while running ``fsstress``.
+* For each scrub item type, create a test to exercise repairing that item type
+  while running ``fsstress``.
+* Race ``fsstress`` and ``xfs_scrub -n`` to ensure that checking the whole
+  filesystem doesn't cause problems.
+* Race ``fsstress`` and ``xfs_scrub`` in force-rebuild mode to ensure that
+  force-repairing the whole filesystem doesn't cause problems. +* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  freezing and thawing the filesystem.
+* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  remounting the filesystem read-only and read-write.
+* The same, but running ``fsx`` instead of ``fsstress``.  (Not done yet?)

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Success is defined by the ability to run all of these tests without
Quoted 4 lines

observing
+any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
+check warnings, or any other sort of mischief.

Seems reasonable. Other than the one nit, I think this section reads pretty well.

Allison

Quoted 10 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Proposed patchsets include `general stress testing
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=race-scrub-and-mount-state-changes>_ +and the evolution of existing per-function stress testing +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=refactor-scrub-stress>`_.

#16 lore raw

Re: [PATCH 03/14] xfs: document the testing plan for online fsck

Quoted 157 lines · Allison Henderson
Quoted 150 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the third chapter of the online fsck design documentation. This
covers the testing plan to make sure that both online and offline fsck
can detect arbitrary problems and correct them without making things worse.

---
 .../filesystems/xfs-online-fsck-design.rst         |  187
++++++++++++++++++++

 1 file changed, 187 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index a03a7b9f0250..d630b6bdbe4a 100644
@@ -563,3 +563,190 @@ functionality.

 Many of these risks are inherent to software programming. Despite this, it is hoped that this new functionality will prove useful in
 reducing unexpected downtime.

Documentation/filesystems/xfs-online-fsck-design.rst +13 -0
+
+3. Testing Plan
+===============
+
+As stated before, fsck tools have three main goals:
+
+1. Detect inconsistencies in the metadata;
+
+2. Eliminate those inconsistencies; and
+
+3. Minimize further loss of data.
+
+Demonstrations of correct operation are necessary to build users'

confidence

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+that the software behaves within expectations.
+Unfortunately, it was not really feasible to perform regular

exhaustive testing
+of every aspect of a fsck tool until the introduction of low-cost virtual

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+machines with high-IOPS storage.
+With ample hardware availability in mind, the testing strategy for

the online
+fsck project involves differential analysis against the existing fsck tools and
+systematic testing of every attribute of every type of metadata object.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+Testing can be split into four major categories, as discussed below.
+
+Integrated Testing with fstests
+-------------------------------
+
+The primary goal of any free software QA effort is to make testing

as
+inexpensive and widespread as possible to maximize the scaling advantages of

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+community.
+In other words, testing should maximize the breadth of filesystem

configuration

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scenarios and hardware setups.
+This improves code quality by enabling the authors of online fsck to

find and
+fix bugs early, and helps developers of new features to find integration

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+issues earlier in their development effort.
+
+The Linux filesystem community shares a common QA testing suite,
+`fstests

<https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/>`_, for

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+functional and regression testing.
+Even before development work began on online fsck, fstests (when run

on XFS)
+would run both the ``xfs_check`` and ``xfs_repair -n`` commands on the test and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+scratch filesystems between each test.
+This provides a level of assurance that the kernel and the fsck

tools stay in

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+alignment about what constitutes consistent metadata.
+During development of the online checking code, fstests was modified

to run
+``xfs_scrub -n`` between each test to ensure that the new checking code

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+produces the same results as the two existing fsck tools.
+
+To start development of online repair, fstests was modified to run
+``xfs_repair`` to rebuild the filesystem's metadata indices between

tests.
+This ensures that offline repair does not crash, leave a corrupt filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+after it exists, or trigger complaints from the online check.
+This also established a baseline for what can and cannot be repaired

offline.
+To complete the first phase of development of online repair, fstests was

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+modified to be able to run ``xfs_scrub`` in a "force rebuild" mode.
+This enables a comparison of the effectiveness of online repair as

compared to

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+the existing offline repair tools.
+
+General Fuzz Testing of Metadata Blocks
+---------------------------------------
+
+XFS benefits greatly from having a very robust debugging tool,

``xfs_db``.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Before development of online fsck even began, a set of fstests were

created
+to test the rather common fault that entire metadata blocks get corrupted.
+This required the creation of fstests library code that can create a filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+containing every possible type of metadata object.
+Next, individual test cases were created to create a test

filesystem, identify
+a single block of a specific type of metadata object, trash it with the
+existing ``blocktrash`` command in ``xfs_db``, and test the reaction of a

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+particular metadata validation strategy.
+
+This earlier test suite enabled XFS developers to test the ability

of the
+in-kernel validation functions and the ability of the offline fsck tool to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+detect and eliminate the inconsistent metadata.
+This part of the test suite was extended to cover online fsck in

exactly the

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+same manner.
+
+In other words, for a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem:
+
+  * Write garbage to it
+
+  * Test the reactions of:
+
+    1. The kernel verifiers to stop obviously bad metadata
+    2. Offline repair (``xfs_repair``) to detect and fix
+    3. Online repair (``xfs_scrub``) to detect and fix
+
+Targeted Fuzz Testing of Metadata Records
+-----------------------------------------
+
+A quick conversation with the other XFS developers revealed that the
Quoted 2 lines

existing
+test infrastructure could be extended to provide

"The testing plan for ofsck includes extending the existing test infrastructure to provide..."

Quoted 1 line

Took me a moment to notice we're not talking about history any more....

Ah. Sorry about that. The sentence now reads:

"The testing plan for online fsck includes extending the existing fs testing infrastructure to provide a much more powerful facility:
targeted fuzz testing of every metadata field of every metadata object in the filesystem."

Quoted 37 lines
Quoted 37 lines

a much more powerful
+facility: targeted fuzz testing of every metadata field of every metadata
+object in the filesystem.
+``xfs_db`` can modify every field of every metadata structure in every
+block in the filesystem to simulate the effects of memory corruption and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+software bugs.
+Given that fstests already contains the ability to create a

filesystem
+containing every metadata format known to the filesystem, ``xfs_db`` can be

Documentation/filesystems/xfs-online-fsck-design.rst +11 -0
+used to perform exhaustive fuzz testing!
+
+For a given fstests filesystem configuration:
+
+* For each metadata object existing on the filesystem...
+
+  * For each record inside that metadata object...
+
+    * For each field inside that record...
+
+      * For each conceivable type of transformation that can be

applied to a bit field...

Documentation/filesystems/xfs-online-fsck-design.rst +12 -0
+
+        1. Clear all bits
+        2. Set all bits
+        3. Toggle the most significant bit
+        4. Toggle the middle bit
+        5. Toggle the least significant bit
+        6. Add a small quantity
+        7. Subtract a small quantity
+        8. Randomize the contents
+
+        * ...test the reactions of:
+
Quoted 8 lines
Quoted 7 lines

+          1. The kernel verifiers to stop obviously bad metadata +          2. Offline checking (``xfs_repair -n``)
+          3. Offline repair (``xfs_repair``)
+          4. Online checking (``xfs_scrub -n``)
+          5. Online repair (``xfs_scrub``)
+          6. Both repair tools (``xfs_scrub`` and then ``xfs_repair`` if online repair doesn't succeed)

I like the indented bullet list format tho

Thanks! I'm pleased that ... whatever renders this stuff ... actually supports nested lists.

Quoted 84 lines
Quoted 77 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+This is quite the combinatoric explosion!
+
+Fortunately, having this much test coverage makes it easy for XFS

developers to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+check the responses of XFS' fsck tools.
+Since the introduction of the fuzz testing framework, these tests

have been
+used to discover incorrect repair code and missing functionality for entire

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+classes of metadata objects in ``xfs_repair``.
+The enhanced testing was used to finalize the deprecation of

``xfs_check`` by
+confirming that ``xfs_repair`` could detect at least as many corruptions as

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the older tool.
+
+These tests have been very valuable for ``xfs_scrub`` in the same

ways -- they
+allow the online fsck developers to compare online fsck against offline fsck,
+and they enable XFS developers to find deficiencies in the code base.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets include
+`general fuzzer improvements
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=fuzzer-improvements>`_,
+`fuzzing baselines
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=fuzz-baseline>`_,
+and `improvements in fuzz testing comprehensiveness +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=more-fuzz-testing>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Stress Testing
+--------------
+
+A unique requirement to online fsck is the ability to operate on a

filesystem

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+concurrently with regular workloads.
+Although it is of course impossible to run ``xfs_scrub`` with *zero*

observable
+impact on the running system, the online repair code should never introduce
+inconsistencies into the filesystem metadata, and regular workloads should

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+never notice resource starvation.
+To verify that these conditions are being met, fstests has been

enhanced in

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+the following ways:
+
+* For each scrub item type, create a test to exercise checking that

item type
+  while running ``fsstress``.
+* For each scrub item type, create a test to exercise repairing that item type
+  while running ``fsstress``.
+* Race ``fsstress`` and ``xfs_scrub -n`` to ensure that checking the whole
+  filesystem doesn't cause problems.
+* Race ``fsstress`` and ``xfs_scrub`` in force-rebuild mode to ensure that
+  force-repairing the whole filesystem doesn't cause problems. +* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  freezing and thawing the filesystem.
+* Race ``xfs_scrub`` in check and force-repair mode against ``fsstress`` while
+  remounting the filesystem read-only and read-write.
+* The same, but running ``fsx`` instead of ``fsstress``.  (Not done yet?)

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Success is defined by the ability to run all of these tests without
Quoted 4 lines

observing
+any unexpected filesystem shutdowns due to corrupted metadata, kernel hang
+check warnings, or any other sort of mischief.

Seems reasonable. Other than the one nit, I think this section reads pretty well.

Quoted 1 line

Woo!

--D

Quoted 12 lines

Allison

Quoted 10 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Proposed patchsets include `general stress testing
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=race-scrub-and-mount-state-changes>_ +and the evolution of existing per-function stress testing +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=refactor-scrub-stress>`_.

#17 lore raw

PATCH 4/14 xfs: document the user interface for online fsck

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the fourth chapter of the online fsck design documentation, which discusses the user interface and the background scrubbing service.

---
 .../filesystems/xfs-online-fsck-design.rst         |  114 ++++++++++++++++++++
 1 file changed, 114 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +114 -0
index d630b6bdbe4a..42e82971e036 100644
@@ -750,3 +750,117 @@ Proposed patchsets include `general stress testing
 <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=race-scrub-and-mount-state-changes>`_
 and the `evolution of existing per-function stress testing
 <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=refactor-scrub-stress>`_.
+
+4. User Interface
+=================
+
+The primary user of online fsck is the system administrator, just like offline
+repair.
+Online fsck presents two modes of operation to administrators:
+A foreground CLI process for online fsck on demand, and a background service
+that performs autonomous checking and repair.
+
+Checking on Demand
+------------------
+
+For administrators who want the absolute freshest information about the
+metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on
+a command line.
+The program checks every piece of metadata in the filesystem while the
+administrator waits for the results to be reported, just like the existing
+``xfs_repair`` tool.
+Both tools share a ``-n`` option to perform a read-only scan, and a ``-v``
+option to increase the verbosity of the information reported.
+
+A new feature of ``xfs_scrub`` is the ``-x`` option, which employs the error
+correction capabilities of the hardware to check data file contents.
+The media scan is not enabled by default because it may dramatically increase
+program runtime and consume a lot of bandwidth on older storage hardware.
+
+The output of a foreground invocation is captured in the system log.
+
+The ``xfs_scrub_all`` program walks the list of mounted filesystems and
+initiates ``xfs_scrub`` for each of them in parallel.
+It serializes scans for any filesystems that resolve to the same top level
+kernel block device to prevent resource overconsumption.
+
+Background Service
+------------------
+
+To reduce the workload of system administrators, the ``xfs_scrub`` package
+provides a suite of `systemd <https://systemd.io/>`_ timers and services that
+run online fsck automatically on weekends.
+The background service configures scrub to run with as little privilege as
+possible, the lowest CPU and IO priority, and in a CPU-constrained single
+threaded mode.
+It is hoped that this minimizes the amount of load generated on the system and
+avoids starving regular workloads.
+
+The output of the background service is also captured in the system log.
+If desired, reports of failures (either due to inconsistencies or mere runtime
+errors) can be emailed automatically by setting the ``EMAIL_ADDR`` environment
+variable in the following service files:
+
+* ``[email protected]``
+* ``[email protected]``
+* ``xfs_scrub_all_fail.service``
+
+The decision to enable the background scan is left to the system administrator.
+This can be done by enabling either of the following services:
+
+* ``xfs_scrub_all.timer`` on systemd systems
+* ``xfs_scrub_all.cron`` on non-systemd systems
+
+This automatic weekly scan is configured out of the box to perform an
+additional media scan of all file data once per month.
+This is less foolproof than, say, storing file data block checksums, but much
+more performant if application software provides its own integrity checking,
+redundancy can be provided elsewhere above the filesystem, or the storage
+device's integrity guarantees are deemed sufficient.
+
+The systemd unit file definitions have been subjected to a security audit
+(as of systemd 249) to ensure that the xfs_scrub processes have as little
+access to the rest of the system as possible.
+This was performed via ``systemd-analyze security``, after which privileges
+were restricted to the minimum required, sandboxing was set up to the maximal
+extent possible with sandboxing and system call filtering; and access to the
+filesystem tree was restricted to the minimum needed to start the program and
+access the filesystem being scanned.
+The service definition files restrict CPU usage to 80% of one CPU core, and
+apply as nice of a priority to IO and CPU scheduling as possible.
+This measure was taken to minimize delays in the rest of the filesystem.
+No such hardening has been performed for the cron job.
+
+Proposed patchset:
+`Enabling the xfs_scrub background service
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-media-scan-service>`_.
+
+Health Reporting
+----------------
+
+XFS caches a summary of each filesystem's health status in memory.
+The information is updated whenever ``xfs_scrub`` is run, or whenever
+inconsistencies are detected in the filesystem metadata during regular
+operations.
+System administrators should use the ``health`` command of ``xfs_spaceman`` to
+download this information into a human-readable format.
+If problems have been observed, the administrator can schedule a reduced
+service window to run the online repair tool to correct the problem.
+Failing that, the administrator can decide to schedule a maintenance window to
+run the traditional offline repair tool to correct the problem.
+
+**Question**: Should the health reporting integrate with the new inotify fs
+error notification system?
+
+**Question**: Would it be helpful for sysadmins to have a daemon to listen for
+corruption notifications and initiate a repair?
+
+*Answer*: These questions remain unanswered, but should be a part of the
+conversation with early adopters and potential downstream users of XFS.
+
+Proposed patchsets include
+`wiring up health reports to correction returns
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports>`_
+and
+`preservation of sickness info during memory reclaim
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_.
#18 lore raw

Re: [PATCH 04/14] xfs: document the user interface for online fsck

Quoted 71 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the fourth chapter of the online fsck design documentation, which
discusses the user interface and the background scrubbing service.

---
 .../filesystems/xfs-online-fsck-design.rst         |  114
++++++++++++++++++++

 1 file changed, 114 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index d630b6bdbe4a..42e82971e036 100644
@@ -750,3 +750,117 @@ Proposed patchsets include `general stress

testing
 <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=race-scrub-and-mount-state-changes>_ and the evolution of existing per-function stress testing <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=refactor-scrub-stress>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+4. User Interface
+=================
+
+The primary user of online fsck is the system administrator, just

like offline

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair.
+Online fsck presents two modes of operation to administrators:
+A foreground CLI process for online fsck on demand, and a background

service

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+that performs autonomous checking and repair.
+
+Checking on Demand
+------------------
+
+For administrators who want the absolute freshest information about

the
+metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+a command line.
+The program checks every piece of metadata in the filesystem while

the
+administrator waits for the results to be reported, just like the existing
+``xfs_repair`` tool.
+Both tools share a ``-n`` option to perform a read-only scan, and a ``-v``

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+option to increase the verbosity of the information reported.
+
+A new feature of ``xfs_scrub`` is the ``-x`` option, which employs

the error

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+correction capabilities of the hardware to check data file contents.
+The media scan is not enabled by default because it may dramatically

increase
+program runtime and consume a lot of bandwidth on older storage hardware.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+The output of a foreground invocation is captured in the system log.
+
+The ``xfs_scrub_all`` program walks the list of mounted filesystems

and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+initiates ``xfs_scrub`` for each of them in parallel.
+It serializes scans for any filesystems that resolve to the same top

level

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include `general stress testing · line 789

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+kernel block device to prevent resource overconsumption.
+
+Background Service
+------------------
+

I'm assuming the below systemd services are configurable right?

Quoted 5 lines

+To reduce the workload of system administrators, the ``xfs_scrub`` package
+provides a suite of systemd <https://systemd.io/>_ timers and services that
+run online fsck automatically on weekends.

by default.

Quoted 5 lines

+The background service configures scrub to run with as little privilege as
+possible, the lowest CPU and IO priority, and in a CPU-constrained single
+threaded mode.

"This can be tuned at anytime to best suit the needs of the customer workload."

Then I think you can drop the below line...

Quoted 89 lines

+It is hoped that this minimizes the amount of load generated on the system and

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+avoids starving regular workloads.
+
+The output of the background service is also captured in the system

log.
+If desired, reports of failures (either due to inconsistencies or mere runtime
+errors) can be emailed automatically by setting the ``EMAIL_ADDR`` environment

Documentation/filesystems/xfs-online-fsck-design.rst +7 -0
+variable in the following service files:
+
+* ``[email protected]``
+* ``[email protected]``
+* ``xfs_scrub_all_fail.service``
+
+The decision to enable the background scan is left to the system

administrator.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+This can be done by enabling either of the following services:
+
+* ``xfs_scrub_all.timer`` on systemd systems
+* ``xfs_scrub_all.cron`` on non-systemd systems
+
+This automatic weekly scan is configured out of the box to perform

an

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+additional media scan of all file data once per month.
+This is less foolproof than, say, storing file data block checksums,

but much
+more performant if application software provides its own integrity checking,
+redundancy can be provided elsewhere above the filesystem, or the storage

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+device's integrity guarantees are deemed sufficient.
+
+The systemd unit file definitions have been subjected to a security

audit
+(as of systemd 249) to ensure that the xfs_scrub processes have as little

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+access to the rest of the system as possible.
+This was performed via ``systemd-analyze security``, after which

privileges
+were restricted to the minimum required, sandboxing was set up to the maximal
+extent possible with sandboxing and system call filtering; and access to the
+filesystem tree was restricted to the minimum needed to start the program and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+access the filesystem being scanned.
+The service definition files restrict CPU usage to 80% of one CPU

core, and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+apply as nice of a priority to IO and CPU scheduling as possible.
+This measure was taken to minimize delays in the rest of the

filesystem.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+No such hardening has been performed for the cron job.
+
+Proposed patchset:
+`Enabling the xfs_scrub background service
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.g
it/log/?h=scrub-media-scan-service>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+Health Reporting
+----------------
+
+XFS caches a summary of each filesystem's health status in memory.
+The information is updated whenever ``xfs_scrub`` is run, or

whenever
+inconsistencies are detected in the filesystem metadata during regular

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+operations.
+System administrators should use the ``health`` command of

``xfs_spaceman`` to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+download this information into a human-readable format.
+If problems have been observed, the administrator can schedule a

reduced

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+service window to run the online repair tool to correct the problem.
+Failing that, the administrator can decide to schedule a maintenance

window to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+run the traditional offline repair tool to correct the problem.
+
+**Question**: Should the health reporting integrate with the new

inotify fs

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+error notification system?
+
+**Question**: Would it be helpful for sysadmins to have a daemon to

listen for

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+corruption notifications and initiate a repair?
+
+*Answer*: These questions remain unanswered, but should be a part of
Quoted 3 lines

the
+conversation with early adopters and potential downstream users of XFS.

I think if there's been no commentary at this point then likely they can't be answered at this time. Perhaps for now it is reasonable to just let the be a potential improvement in the future if the demand for it arises. In any case, I think we should probably clean out the Q&A discussion prompts.

Rest looks good tho
Allison

Quoted 12 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets include
+`wiring up health reports to correction returns
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=corruption-health-reports>`_
+and
+`preservation of sickness info during memory reclaim +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=indirect-health-reporting>`_.

#19 lore raw

Re: [PATCH 04/14] xfs: document the user interface for online fsck

Quoted 70 lines · Allison Henderson
Quoted 69 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Start the fourth chapter of the online fsck design documentation, which
discusses the user interface and the background scrubbing service.

---
 .../filesystems/xfs-online-fsck-design.rst         |  114
++++++++++++++++++++

 1 file changed, 114 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index d630b6bdbe4a..42e82971e036 100644
@@ -750,3 +750,117 @@ Proposed patchsets include `general stress

testing
 <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=race-scrub-and-mount-state-changes>_ and the evolution of existing per-function stress testing <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.g
it/log/?h=refactor-scrub-stress>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+4. User Interface
+=================
+
+The primary user of online fsck is the system administrator, just

like offline

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+repair.
+Online fsck presents two modes of operation to administrators:
+A foreground CLI process for online fsck on demand, and a background

service

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+that performs autonomous checking and repair.
+
+Checking on Demand
+------------------
+
+For administrators who want the absolute freshest information about

the
+metadata in a filesystem, ``xfs_scrub`` can be run as a foreground process on

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+a command line.
+The program checks every piece of metadata in the filesystem while

the
+administrator waits for the results to be reported, just like the existing
+``xfs_repair`` tool.
+Both tools share a ``-n`` option to perform a read-only scan, and a ``-v``

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+option to increase the verbosity of the information reported.
+
+A new feature of ``xfs_scrub`` is the ``-x`` option, which employs

the error

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+correction capabilities of the hardware to check data file contents.
+The media scan is not enabled by default because it may dramatically

increase
+program runtime and consume a lot of bandwidth on older storage hardware.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+The output of a foreground invocation is captured in the system log.
+
+The ``xfs_scrub_all`` program walks the list of mounted filesystems

and
+initiates ``xfs_scrub`` for each of them in parallel.

Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include `general stress testing · line 789

Quoted 7 lines

+It serializes scans for any filesystems that resolve to the same top level

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+kernel block device to prevent resource overconsumption.
+
+Background Service
+------------------
+

I'm assuming the below systemd services are configurable right?

Yes, through the standard systemd overriddes.

Quoted 6 lines
Quoted 5 lines

+To reduce the workload of system administrators, the ``xfs_scrub`` package
+provides a suite of systemd <https://systemd.io/>_ timers and services that
+run online fsck automatically on weekends.

by default.

Fixed.

Quoted 7 lines
Quoted 5 lines

+The background service configures scrub to run with as little privilege as
+possible, the lowest CPU and IO priority, and in a CPU-constrained single
+threaded mode.

"This can be tuned at anytime to best suit the needs of the customer workload."

Fixed.

Quoted 4 lines

Then I think you can drop the below line...

Quoted 3 lines

+It is hoped that this minimizes the amount of load generated on the system and
+avoids starving regular workloads.

Done.

Quoted 85 lines
Quoted 85 lines

+The output of the background service is also captured in the system log.
+If desired, reports of failures (either due to inconsistencies or mere runtime
+errors) can be emailed automatically by setting the ``EMAIL_ADDR`` environment

Documentation/filesystems/xfs-online-fsck-design.rst +7 -0
+variable in the following service files:
+
+* ``[email protected]``
+* ``[email protected]``
+* ``xfs_scrub_all_fail.service``
+
+The decision to enable the background scan is left to the system

administrator.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+This can be done by enabling either of the following services:
+
+* ``xfs_scrub_all.timer`` on systemd systems
+* ``xfs_scrub_all.cron`` on non-systemd systems
+
+This automatic weekly scan is configured out of the box to perform

an

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+additional media scan of all file data once per month.
+This is less foolproof than, say, storing file data block checksums,

but much
+more performant if application software provides its own integrity checking,
+redundancy can be provided elsewhere above the filesystem, or the storage

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+device's integrity guarantees are deemed sufficient.
+
+The systemd unit file definitions have been subjected to a security

audit
+(as of systemd 249) to ensure that the xfs_scrub processes have as little

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+access to the rest of the system as possible.
+This was performed via ``systemd-analyze security``, after which

privileges
+were restricted to the minimum required, sandboxing was set up to the maximal
+extent possible with sandboxing and system call filtering; and access to the
+filesystem tree was restricted to the minimum needed to start the program and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+access the filesystem being scanned.
+The service definition files restrict CPU usage to 80% of one CPU

core, and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+apply as nice of a priority to IO and CPU scheduling as possible.
+This measure was taken to minimize delays in the rest of the

filesystem.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+No such hardening has been performed for the cron job.
+
+Proposed patchset:
+`Enabling the xfs_scrub background service
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.g
it/log/?h=scrub-media-scan-service>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+Health Reporting
+----------------
+
+XFS caches a summary of each filesystem's health status in memory.
+The information is updated whenever ``xfs_scrub`` is run, or

whenever
+inconsistencies are detected in the filesystem metadata during regular

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+operations.
+System administrators should use the ``health`` command of

``xfs_spaceman`` to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+download this information into a human-readable format.
+If problems have been observed, the administrator can schedule a

reduced

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+service window to run the online repair tool to correct the problem.
+Failing that, the administrator can decide to schedule a maintenance

window to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+run the traditional offline repair tool to correct the problem.
+
+**Question**: Should the health reporting integrate with the new

inotify fs

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+error notification system?
+
+**Question**: Would it be helpful for sysadmins to have a daemon to

listen for

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+corruption notifications and initiate a repair?
+
+*Answer*: These questions remain unanswered, but should be a part of
Quoted 8 lines
Quoted 3 lines

the
+conversation with early adopters and potential downstream users of XFS.

I think if there's been no commentary at this point then likely they can't be answered at this time. Perhaps for now it is reasonable to just let the be a potential improvement in the future if the demand for it arises. In any case, I think we should probably clean out the Q&A discussion prompts.

I'll change them to "future work Q's" so I don't forget to pursue them after part 1 is merged.

Quoted 1 line

Rest looks good tho

:-D Thanks!

--D

Quoted 15 lines

Allison

Quoted 12 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets include
+`wiring up health reports to correction returns
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=corruption-health-reports>`_
+and
+`preservation of sickness info during memory reclaim +<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=indirect-health-reporting>`_.

#20 lore raw

PATCH 5/14 xfs: document the filesystem metadata checking strategy

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Begin the fifth chapter of the online fsck design documentation, where we discuss the details of the data structures and algorithms used by the kernel to examine filesystem metadata and cross-reference it around the filesystem.

---
 .../filesystems/xfs-online-fsck-design.rst         |  579 ++++++++++++++++++++
 .../filesystems/xfs-self-describing-metadata.rst   |    1 
 2 files changed, 580 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +579 -0
index 42e82971e036..f45bf97fa9c4 100644
@@ -864,3 +864,582 @@ Proposed patchsets include
 and
 `preservation of sickness info during memory reclaim
 <https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting>`_.
+
+5. Kernel Algorithms and Data Structures
+========================================
+
+This section discusses the key algorithms and data structures of the kernel
+code that provide the ability to check and repair metadata while the system
+is running.
+The first chapters in this section reveal the pieces that provide the
+foundation for checking metadata.
+The remainder of this section presents the mechanisms through which XFS
+regenerates itself.
+
+Self Describing Metadata
+------------------------
+
+Starting with XFS version 5 in 2012, XFS updated the format of nearly every
+ondisk block header to record a magic number, a checksum, a universally
+"unique" identifier (UUID), an owner code, the ondisk address of the block,
+and a log sequence number.
+When loading a block buffer from disk, the magic number, UUID, owner, and
+ondisk address confirm that the retrieved block matches the specific owner of
+the current filesystem, and that the information contained in the block is
+supposed to be found at the ondisk address.
+The first three components enable checking tools to disregard alleged metadata
+that doesn't belong to the filesystem, and the fourth component enables the
+filesystem to detect lost writes.
+
+The logging code maintains the checksum and the log sequence number of the last
+transactional update.
+Checksums are useful for detecting torn writes and other mischief between the
+computer and its storage devices.
+Sequence number tracking enables log recovery to avoid applying out of date
+log updates to the filesystem.
+
+These two features improve overall runtime resiliency by providing a means for
+the filesystem to detect obvious corruption when reading metadata blocks from
+disk, but these buffer verifiers cannot provide any consistency checking
+between metadata structures.
+
+For more information, please see the documentation for
+Documentation/filesystems/xfs-self-describing-metadata.rst
+
+Reverse Mapping
+---------------
+
+The original design of XFS (circa 1993) is an improvement upon 1980s Unix
+filesystem design.
+In those days, storage density was expensive, CPU time was scarce, and
+excessive seek time could kill performance.
+For performance reasons, filesystem authors were reluctant to add redundancy to
+the filesystem, even at the cost of data integrity.
+Filesystems designers in the early 21st century choose different strategies to
+increase internal redundancy -- either storing nearly identical copies of
+metadata, or more space-efficient techniques such as erasure coding.
+Obvious corruptions are typically repaired by copying replicas or
+reconstructing from codes.
+
+For XFS, a different redundancy strategy was chosen to modernize the design:
+a secondary space usage index that maps allocated disk extents back to their
+owners.
+By adding a new index, the filesystem retains most of its ability to scale
+well to heavily threaded workloads involving large datasets, since the primary
+file metadata (the directory tree, the file block map, and the allocation
+groups) remain unchanged.
+Although the reverse-mapping feature increases overhead costs for space
+mapping activities just like any other system that improves redundancy, it
+has two critical advantages: first, the reverse index is key to enabling online
+fsck and other requested functionality such as filesystem reorganization,
+better media failure reporting, and shrinking.
+Second, the different ondisk storage format of the reverse mapping btree
+defeats device-level deduplication, because the filesystem requires real
+redundancy.
+
+A criticism of adding the secondary index is that it does nothing to improve
+the robustness of user data storage itself.
+This is a valid point, but adding a new index for file data block checksums
+increases write amplification and turns data overwrites into copy-writes, which
+age the filesystem prematurely.
+In keeping with thirty years of precedent, users who want file data integrity
+can supply as powerful a solution as they require.
+As for metadata, the complexity of adding a new secondary index of space usage
+is much less than adding volume management and storage device mirroring to XFS
+itself.
+Perfection of RAID and volume management are best left to existing layers in
+the kernel.
+
+The information captured in a reverse space mapping record is as follows:
+
+.. code-block:: c
+
+	struct xfs_rmap_irec {
+	    xfs_agblock_t    rm_startblock;   /* extent start block */
+	    xfs_extlen_t     rm_blockcount;   /* extent length */
+	    uint64_t         rm_owner;        /* extent owner */
+	    uint64_t         rm_offset;       /* offset within the owner */
+	    unsigned int     rm_flags;        /* state flags */
+	};
+
+The first two fields capture the location and size of the physical space,
+in units of filesystem blocks.
+The owner field tells scrub which metadata structure or file inode have been
+assigned this space.
+For space allocated to files, the offset field tells scrub where the space was
+mapped within the file fork.
+Finally, the flags field provides extra information about the space usage --
+is this an attribute fork extent?  A file mapping btree extent?  Or an
+unwritten data extent?
+
+Online filesystem checking judges the consistency of each primary metadata
+record by comparing its information against all other space indices.
+The reverse mapping index plays a key role in the consistency checking process
+because it contains a centralized alternate copy of all space allocation
+information.
+Program runtime and ease of resource acquisition are the only real limits to
+what online checking can consult.
+For example, a file data extent mapping can be checked against:
+
+* The absence of an entry in the free space information.
+* The absence of an entry in the inode index.
+* The absence of an entry in the reference count data if the file is not
+  marked as having shared extents.
+* The correspondence of an entry in the reverse mapping information.
+
+A key observation here is that only the reverse mapping can provide a positive
+affirmation of correctness if the primary metadata is in doubt.
+The checking code for most primary metadata follows a path similar to the
+one outlined above.
+
+A second observation to make about this secondary index is that proving its
+consistency with the primary metadata is difficult.
+Demonstrating that a given reverse mapping record exactly corresponds to the
+primary space metadata involves a full scan of all primary space metadata,
+which is very time intensive.
+Scanning activity for online fsck can only use non-blocking lock acquisition
+primitives if the locking order is not the regular order as used by the rest of
+the filesystem.
+This means that forward progress during this part of a scan of the reverse
+mapping data cannot be guaranteed if system load is especially heavy.
+Therefore, it is not practical for online check to detect reverse mapping
+records that lack a counterpart in the primary metadata.
+Instead, scrub relies on rigorous cross-referencing during the primary space
+mapping structure checks.
+
+Reverse mappings also play a key role in reconstruction of primary metadata.
+The secondary information is general enough for online repair to synthesize a
+complete copy of any primary space management metadata by locking that
+resource, querying all reverse mapping indices looking for records matching
+the relevant resource, and transforming the mapping into an appropriate format.
+The details of how these records are staged, written to disk, and committed
+into the filesystem are covered in subsequent sections.
+
+Checking and Cross-Referencing
+------------------------------
+
+The first step of checking a metadata structure is to examine every record
+contained within the structure and its relationship with the rest of the
+system.
+XFS contains multiple layers of checking to try to prevent inconsistent
+metadata from wreaking havoc on the system.
+Each of these layers contributes information that helps the kernel to make
+three decisions about the health of a metadata structure:
+
+- Is a part of this structure obviously corrupt (``XFS_SCRUB_OFLAG_CORRUPT``) ?
+- Is this structure inconsistent with the rest of the system
+  (``XFS_SCRUB_OFLAG_XCORRUPT``) ?
+- Is there so much damage around the filesystem that cross-referencing is not
+  possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
+- Can the structure be optimized to improve performance or reduce the size of
+  metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
+- Does the structure contain data that is not inconsistent but deserves review
+  by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?
+
+The following sections describe how the metadata scrubbing process works.
+
+Metadata Buffer Verification
+````````````````````````````
+
+The lowest layer of metadata protection in XFS are the metadata verifiers built
+into the buffer cache.
+These functions perform inexpensive internal consistency checking of the block
+itself, and answer these questions:
+
+- Does the block belong to this filesystem?
+
+- Does the block belong to the structure that asked for the read?
+  This assumes that metadata blocks only have one owner, which is always true
+  in XFS.
+
+- Is the type of data stored in the block within a reasonable range of what
+  scrub is expecting?
+
+- Does the physical location of the block match the location it was read from?
+
+- Does the block checksum match the data?
+
+The scope of the protections here are very limited -- verifiers can only
+establish that the filesystem code is reasonably free of gross corruption bugs
+and that the storage system is reasonably competent at retrieval.
+Corruption problems observed at runtime cause the generation of health reports,
+failed system calls, and in the extreme case, filesystem shutdowns if the
+corrupt metadata force the cancellation of a dirty transaction.
+
+Every online fsck scrubbing function is expected to read every ondisk metadata
+block of a structure in the course of checking the structure.
+Corruption problems observed during a check are immediately reported to
+userspace as corruption; during a cross-reference, they are reported as a
+failure to cross-reference once the full examination is complete.
+Reads satisfied by a buffer already in cache (and hence already verified)
+bypass these checks.
+
+Internal Consistency Checks
+```````````````````````````
+
+The next higher level of metadata protection is the internal record
+verification code built into the filesystem.
+These checks are split between the buffer verifiers, the in-filesystem users of
+the buffer cache, and the scrub code itself, depending on the amount of higher
+level context required.
+The scope of checking is still internal to the block.
+For performance reasons, regular code may skip some of these checks unless
+debugging is enabled or a write is about to occur.
+Scrub functions, of course, must check all possible problems.
+Either way, these higher level checking functions answer these questions:
+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the read?
+
+- If the block contains records, do the records fit within the block?
+
+- If the block tracks internal free space information, is it consistent with
+  the record areas?
+
+- Are the records contained inside the block free of obvious corruptions?
+
+Record checks in this category are more rigorous and more time-intensive.
+For example, block pointers and inumbers are checked to ensure that they point
+within the dynamically allocated parts of an allocation group and within
+the filesystem.
+Names are checked for invalid characters, and flags are checked for invalid
+combinations.
+Other record attributes are checked for sensible values.
+Btree records spanning an interval of the btree keyspace are checked for
+correct order and lack of mergeability (except for file fork mappings).
+
+Validation of Userspace-Controlled Record Attributes
+````````````````````````````````````````````````````
+
+Various pieces of filesystem metadata are directly controlled by userspace.
+Because of this nature, validation work cannot be more precise than checking
+that a value is within the possible range.
+These fields include:
+
+- Superblock fields controlled by mount options
+- Filesystem labels
+- File timestamps
+- File permissions
+- File size
+- File flags
+- Names present in directory entries, extended attribute keys, and filesystem
+  labels
+- Extended attribute key namespaces
+- Extended attribute values
+- File data block contents
+- Quota limits
+- Quota timer expiration (if resource usage exceeds the soft limit)
+
+Cross-Referencing Space Metadata
+````````````````````````````````
+
+The next higher level of checking is cross-referencing records between metadata
+structures.
+For regular runtime code, the cost of these checks is considered to be
+prohibitively expensive, but as scrub is dedicated to rooting out
+inconsistencies, it must pursue all avenues of inquiry.
+The exact set of cross-referencing is highly dependent on the context of the
+data structure being checked.
+
+The XFS btree code has keyspace scanning functions that online fsck uses to
+cross reference one structure with another.
+Specifically, scrub can scan the key space of an index to determine if that
+keyspace is fully, sparsely, or not at all mapped to records.
+For the reverse mapping btree, it is possible to mask parts of the key for the
+purposes of performing a keyspace scan so that scrub can decide if the rmap
+btree contains records mapping a certain extent of physical space without the
+sparsenses of the rest of the rmap keyspace getting in the way.
+
+Btree blocks undergo the following checks before cross-referencing:
+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the read?
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious corruptions?
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the btree point to valid block addresses for the type
+  of btree?
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each node block record, does the record key accurate reflect the contents
+  of the child block?
+
+Space allocation records are cross-referenced as follows:
+
+1. Any space mentioned by any metadata structure are cross-referenced as
+   follows:
+
+   - Does the reverse mapping index list only the appropriate owner as the
+     owner of each block?
+
+   - Are none of the blocks claimed as free space?
+
+   - If these aren't file data blocks, are none of the blocks claimed as space
+     shared by different owners?
+
+2. Btree blocks are cross-referenced as follows:
+
+   - Everything in class 1 above.
+
+   - If there's a parent node block, do the keys listed for this block match the
+     keyspace of this block?
+
+   - Do the sibling pointers point to valid blocks?  Of the same level?
+
+   - Do the child pointers point to valid blocks?  Of the next level down?
+
+3. Free space btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Does the reverse mapping index list no owners of this space?
+
+   - Is this space not claimed by the inode index for inodes?
+
+   - Is it not mentioned by the reference count index?
+
+   - Is there a matching record in the other free space btree?
+
+4. Inode btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is there a matching record in free inode btree?
+
+   - Do cleared bits in the holemask correspond with inode clusters?
+
+   - Do set bits in the freemask correspond with inode records with zero link
+     count?
+
+5. Inode records are cross-referenced as follows:
+
+   - Everything in class 1.
+
+   - Do all the fields that summarize information about the file forks actually
+     match those forks?
+
+   - Does each inode with zero link count correspond to a record in the free
+     inode btree?
+
+6. File fork space mapping records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is this space not mentioned by the inode btrees?
+
+   - If this is a CoW fork mapping, does it correspond to a CoW entry in the
+     reference count btree?
+
+7. Reference count records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Within the space subkeyspace of the rmap btree (that is to say, all
+     records mapped to a particular space extent and ignoring the owner info),
+     are there the same number of reverse mapping records for each block as the
+     reference count record claims?
+
+Proposed patchsets are the series to find gaps in
+`refcount btree
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-refcount-gaps>`_,
+`inode btree
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-inobt-gaps>`_, and
+`rmap btree
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-rmapbt-gaps>`_ records;
+to find
+`mergeable records
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-detect-mergeable-records>`_;
+and to
+`improve cross referencing with rmap
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-strengthen-rmap-checking>`_
+before starting a repair.
+
+Checking Extended Attributes
+````````````````````````````
+
+Extended attributes implement a key-value store that enable fragments of data
+to be attached to any file.
+Both the kernel and userspace can access the keys and values, subject to
+namespace and privilege restrictions.
+Most typically these fragments are metadata about the file -- origins, security
+contexts, user-supplied labels, indexing information, etc.
+
+Names can be as long as 255 bytes and can exist in several different
+namespaces.
+Values can be as large as 64KB.
+A file's extended attributes are stored in blocks mapped by the attr fork.
+The mappings point to leaf blocks, remote value blocks, or dabtree blocks.
+Block 0 in the attribute fork is always the top of the structure, but otherwise
+each of the three types of blocks can be found at any offset in the attr fork.
+Leaf blocks contain attribute key records that point to the name and the value.
+Names are always stored elsewhere in the same leaf block.
+Values that are less than 3/4 the size of a filesystem block are also stored
+elsewhere in the same leaf block.
+Remote value blocks contain values that are too large to fit inside a leaf.
+If the leaf information exceeds a single filesystem block, a dabtree (also
+rooted at block 0) is created to map hashes of the attribute names to leaf
+blocks in the attr fork.
+
+Checking an extended attribute structure is not so straightfoward due to the
+lack of separation between attr blocks and index blocks.
+Scrub must read each block mapped by the attr fork and ignore the non-leaf
+blocks:
+
+1. Walk the dabtree in the attr fork (if present) to ensure that there are no
+   irregularities in the blocks or dabtree mappings that do not point to
+   attr leaf blocks.
+
+2. Walk the blocks of the attr fork looking for leaf blocks.
+   For each entry inside a leaf:
+
+   a. Validate that the name does not contain invalid characters.
+
+   b. Read the attr value.
+      This performs a named lookup of the attr name to ensure the correctness
+      of the dabtree.
+      If the value is stored in a remote block, this also validates the
+      integrity of the remote value block.
+
+Checking and Cross-Referencing Directories
+``````````````````````````````````````````
+
+The filesystem directory tree is a directed acylic graph structure, with files
+constituting the nodes, and directory entries (dirents) constituting the edges.
+Directories are a special type of file containing a set of mappings from a
+255-byte sequence (name) to an inumber.
+These are called directory entries, or dirents for short.
+Each directory file must have exactly one directory pointing to the file.
+A root directory points to itself.
+Directory entries point to files of any type.
+Each non-directory file may have multiple directories point to it.
+
+In XFS, directories are implemented as a file containing up to three 32GB
+partitions.
+The first partition contains directory entry data blocks.
+Each data block contains variable-sized records associating a user-provided
+name with an inumber and, optionally, a file type.
+If the directory entry data grows beyond one block, the second partition (which
+exists as post-EOF extents) is populated with a block containing free space
+information and an index that maps hashes of the dirent names to directory data
+blocks in the first partition.
+This makes directory name lookups very fast.
+If this second partition grows beyond one block, the third partition is
+populated with a linear array of free space information for faster
+expansions.
+If the free space has been separated and the second partition grows again
+beyond one block, then a dabtree is used to map hashes of dirent names to
+directory data blocks.
+
+Checking a directory is pretty straightfoward:
+
+1. Walk the dabtree in the second partition (if present) to ensure that there
+   are no irregularities in the blocks or dabtree mappings that do not point to
+   dirent blocks.
+
+2. Walk the blocks of the first partition looking for directory entries.
+   Each dirent is checked as follows:
+
+   a. Does the name contain no invalid characters?
+
+   b. Does the inumber correspond to an actual, allocated inode?
+
+   c. Does the child inode have a nonzero link count?
+
+   d. If a file type is included in the dirent, does it match the type of the
+      inode?
+
+   e. If the child is a subdirectory, does the child's dotdot pointer point
+      back to the parent?
+
+   f. If the directory has a second partition, perform a named lookup of the
+      dirent name to ensure the correctness of the dabtree.
+
+3. Walk the free space list in the third partition (if present) to ensure that
+   the free spaces it describes are really unused.
+
+Checking operations involving :ref:`parents <dirparent>` and
+:ref:`file link counts <nlinks>` are discussed in more detail in later
+sections.
+
+Checking Directory/Attribute Btrees
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As stated in previous sections, the directory/attribute btree (dabtree) index
+maps user-provided names to improve lookup times by avoiding linear scans.
+Internally, it maps a 32-bit hash of the name to a block offset within the
+appropriate file fork.
+
+The internal structure of a dabtree closely resembles the btrees that record
+fixed-size metadata records -- each dabtree block contains a magic number, a
+checksum, sibling pointers, a UUID, a tree level, and a log sequence number.
+The format of leaf and node records are the same -- each entry points to the
+next level down in the hierarchy, with dabtree node records pointing to dabtree
+leaf blocks, and dabtree leaf records pointing to non-dabtree blocks elsewhere
+in the fork.
+
+Checking and cross-referencing the dabtree is very similar to what is done for
+space btrees:
+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the read?
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious corruptions?
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the dabtree point to valid fork offsets for dabtree
+  blocks?
+
+- Do leaf pointers within the dabtree point to valid fork offsets for directory
+  or attr leaf blocks?
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each dabtree node record, does the record key accurate reflect the
+  contents of the child dabtree block?
+
+- For each dabtree leaf record, does the record key accurate reflect the
+  contents of the directory or attr block?
+
+Cross-Referencing Summary Counters
+``````````````````````````````````
+
+XFS maintains three classes of summary counters: available resources, quota
+resource usage, and file link counts.
+
+In theory, the amount of available resources (data blocks, inodes, realtime
+extents) can be found by walking the entire filesystem.
+This would make for very slow reporting, so a transactional filesystem can
+maintain summaries of this information in the superblock.
+Cross-referencing these values against the filesystem metadata should be a
+simple matter of walking the free space and inode metadata in each AG and the
+realtime bitmap, but there are complications that will be discussed in
+:ref:`more detail <fscounters>` later.
+
+:ref:`Quota usage <quotacheck>` and :ref:`file link count <nlinks>`
+checking are sufficiently complicated to warrant separate sections.
+
+Post-Repair Reverification
+``````````````````````````
+
+After performing a repair, the checking code is run a second time to validate
+the new structure, and the results of the health assessment are recorded
+internally and returned to the calling process.
+This step is critical for enabling system administrator to monitor the status
+of the filesystem and the progress of any repairs.
+For developers, it is a useful means to judge the efficacy of error detection
+and correction in the online and offline checking tools.
Documentation/filesystems/xfs-self-describing-metadata.rst +1 -0
index b79dbf36dc94..a10c4ae6955e 100644
@@ -1,4 +1,5 @@
 .. SPDX-License-Identifier: GPL-2.0
+.. _xfs_self_describing_metadata:
 
 ============================
 XFS Self Describing Metadata
#21 lore raw

Re: [PATCH 05/14] xfs: document the filesystem metadata checking strategy

Quoted 63 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Begin the fifth chapter of the online fsck design documentation, where
we discuss the details of the data structures and algorithms used by the
kernel to examine filesystem metadata and cross-reference it around the
filesystem.

---
 .../filesystems/xfs-online-fsck-design.rst         |  579
++++++++++++++++++++
 .../filesystems/xfs-self-describing-metadata.rst   |    1

 2 files changed, 580 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 42e82971e036..f45bf97fa9c4 100644
@@ -864,3 +864,582 @@ Proposed patchsets include

 and
 `preservation of sickness info during memory reclaim <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=indirect-health-reporting>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+5. Kernel Algorithms and Data Structures
+========================================
+
+This section discusses the key algorithms and data structures of the

kernel
+code that provide the ability to check and repair metadata while the system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+is running.
+The first chapters in this section reveal the pieces that provide

the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+foundation for checking metadata.
+The remainder of this section presents the mechanisms through which

XFS

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+regenerates itself.
+
+Self Describing Metadata
+------------------------
+
+Starting with XFS version 5 in 2012, XFS updated the format of

nearly every
+ondisk block header to record a magic number, a checksum, a universally
+"unique" identifier (UUID), an owner code, the ondisk address of the block,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+and a log sequence number.
+When loading a block buffer from disk, the magic number, UUID,

owner, and
+ondisk address confirm that the retrieved block matches the specific owner of
+the current filesystem, and that the information contained in the block is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+supposed to be found at the ondisk address.
+The first three components enable checking tools to disregard
Quoted 4 lines

alleged metadata
+that doesn't belong to the filesystem, and the fourth component enables the
+filesystem to detect lost writes.

Add...

"When ever a file system operation modifies a block, the change is submitted to the journal as a transaction. The journal then processes these transactions marking them done once they are safely committed to the disk"

At this point we havnt talked much at all about transactions or logs, and we've just barely begin to cover blocks. I think you at least want a quick blip to describe the relation of these two things, or it may not be clear why we suddenly jumped into logs.

Documentation/filesystems/xfs-online-fsck-design.rst

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The logging code maintains the checksum and the log sequence number

of the last

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+transactional update.
+Checksums are useful for detecting torn writes and other mischief

"Checksums (or crc's) are useful for detecting incomplete or torn writes as well as other discrepancies..."

Quoted 30 lines

between the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer and its storage devices.
+Sequence number tracking enables log recovery to avoid applying out

of date

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+log updates to the filesystem.
+
+These two features improve overall runtime resiliency by providing a

means for
+the filesystem to detect obvious corruption when reading metadata blocks from
+disk, but these buffer verifiers cannot provide any consistency checking

Documentation/filesystems/xfs-online-fsck-design.rst +9 -0
+between metadata structures.
+
+For more information, please see the documentation for
+Documentation/filesystems/xfs-self-describing-metadata.rst
+
+Reverse Mapping
+---------------
+
+The original design of XFS (circa 1993) is an improvement upon 1980s

Unix

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem design.
+In those days, storage density was expensive, CPU time was scarce,

and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+excessive seek time could kill performance.
+For performance reasons, filesystem authors were reluctant to add

redundancy to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem, even at the cost of data integrity.
+Filesystems designers in the early 21st century choose different
Quoted 4 lines

strategies to
+increase internal redundancy -- either storing nearly identical copies of
+metadata, or more space-efficient techniques such as erasure coding.

"such as erasure coding which may encode sections of the data with redundant symbols and in more than one location"

That ties it into the next line. If you go on to talk about a term you have not previously defined, i think you want to either define it quickly or just drop it all together. Right now your goal is to just give the reader context, so you want it to move quickly.

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 923

Quoted 3 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+Obvious corruptions are typically repaired by copying replicas or
+reconstructing from codes.
+

I think I would have just jumped straight from xfs history to modern xfs...

Quoted 15 lines

+For XFS, a different redundancy strategy was chosen to modernize the design:
+a secondary space usage index that maps allocated disk extents back to their

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+owners.
+By adding a new index, the filesystem retains most of its ability to

scale
+well to heavily threaded workloads involving large datasets, since the primary
+file metadata (the directory tree, the file block map, and the allocation

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+groups) remain unchanged.
 
 
+Although the reverse-mapping feature increases overhead costs for
Quoted 3 lines

space
+mapping activities just like any other system that improves redundancy, it

"Like any system that improves redundancy, the reverse-mapping feature increases overhead costs for space mapping activities. However, it..."

Quoted 28 lines

+has two critical advantages: first, the reverse index is key to enabling online
+fsck and other requested functionality such as filesystem reorganization,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+better media failure reporting, and shrinking.
+Second, the different ondisk storage format of the reverse mapping

btree
+defeats device-level deduplication, because the filesystem requires real

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+redundancy.
+
+A criticism of adding the secondary index is that it does nothing to

improve

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the robustness of user data storage itself.
+This is a valid point, but adding a new index for file data block

checksums
+increases write amplification and turns data overwrites into copy- writes, which

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+age the filesystem prematurely.
+In keeping with thirty years of precedent, users who want file data

integrity

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+can supply as powerful a solution as they require.
+As for metadata, the complexity of adding a new secondary index of

space usage
+is much less than adding volume management and storage device mirroring to XFS

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+itself.
+Perfection of RAID and volume management are best left to existing
Quoted 2 lines

layers in
+the kernel.

I think I would cull the entire above paragraph. rmap, crc and raid all have very different points of redundancy, so criticism that an apple is not an orange or visavis just feels like a shortsighted comparison that's probably more of a distraction than anything.

Sometimes it feels like this document kinda gets off into tangents like it's preemptively trying to position it's self for an argument that hasn't happened yet. But I think it has the effect of pulling the readers attention off topic into an argument they never thought to consider in the first place. The topic of this section is to explain what rmap is. So lets stay on topic and finish laying out that ground work first before getting into how it compares to other solutions

Quoted 52 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The information captured in a reverse space mapping record is as

follows:

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+.. code-block:: c
+
+       struct xfs_rmap_irec {
+           xfs_agblock_t    rm_startblock;   /* extent start block

*/
+           xfs_extlen_t     rm_blockcount;   /* extent length */ +           uint64_t         rm_owner;        /* extent owner */ +           uint64_t         rm_offset;       /* offset within the owner */
+           unsigned int     rm_flags;        /* state flags */ +       };

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The first two fields capture the location and size of the physical

space,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in units of filesystem blocks.
+The owner field tells scrub which metadata structure or file inode

have been

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+assigned this space.
+For space allocated to files, the offset field tells scrub where the

space was

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+mapped within the file fork.
+Finally, the flags field provides extra information about the space

usage --
+is this an attribute fork extent?  A file mapping btree extent?  Or an

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+unwritten data extent?
+
+Online filesystem checking judges the consistency of each primary

metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+record by comparing its information against all other space indices.
+The reverse mapping index plays a key role in the consistency

checking process
+because it contains a centralized alternate copy of all space allocation

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+information.
+Program runtime and ease of resource acquisition are the only real

limits to

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+what online checking can consult.
+For example, a file data extent mapping can be checked against:
+
+* The absence of an entry in the free space information.
+* The absence of an entry in the inode index.
+* The absence of an entry in the reference count data if the file is

not
+  marked as having shared extents.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+* The correspondence of an entry in the reverse mapping information.
+
+A key observation here is that only the reverse mapping can provide
Quoted 2 lines

a positive
+affirmation of correctness if the primary metadata is in doubt.

if any of the above metadata is in doubt...

Quoted 9 lines

+The checking code for most primary metadata follows a path similar to the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+one outlined above.
+
+A second observation to make about this secondary index is that

proving its

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency with the primary metadata is difficult.
 
+Demonstrating that a given reverse mapping record exactly
Quoted 4 lines

corresponds to the
+primary space metadata involves a full scan of all primary space metadata,
+which is very time intensive.

"But why?" Wonders the reader. Just jump into an example:

"In order to verify that an rmap extent does not incorrectly over lap with another record, we would need a full scan of all the other records, which is time intensive."

?

And then the below is a separate observation right?

Quoted 5 lines

+Scanning activity for online fsck can only use non-blocking lock acquisition
+primitives if the locking order is not the regular order as used by the rest of
+the filesystem.

Lastly, it should be noted that most file system operations tend to lock primary metadata before locking the secondary metadata. This means that scanning operations that acquire the secondary metadata first may need to yield the secondary lock to filesystem operations that have already acquired the primary lock.

?

Quoted 7 lines

+This means that forward progress during this part of a scan of the reverse
+mapping data cannot be guaranteed if system load is especially heavy.
+Therefore, it is not practical for online check to detect reverse mapping
+records that lack a counterpart in the primary metadata.

Such as <quick list / quick example>

Documentation/filesystems/xfs-online-fsck-design.rst

Quoted 4 lines

+Instead, scrub relies on rigorous cross-referencing during the primary space

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+mapping structure checks.
+

The below paragraph sounds like a re-cap?

"So to recap, reverse mappings also...."

Quoted 5 lines

+Reverse mappings also play a key role in reconstruction of primary metadata.
+The secondary information is general enough for online repair to synthesize a
+complete copy of any primary space management metadata by locking

Quoted 8 lines

that
+resource, querying all reverse mapping indices looking for records matching
+the relevant resource, and transforming the mapping into an appropriate format.
+The details of how these records are staged, written to disk, and committed
+into the filesystem are covered in subsequent sections.

I also think the section would be ok if you were to trim off this last paragraph too.

Quoted 80 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing
+------------------------------
+
+The first step of checking a metadata structure is to examine every

record
+contained within the structure and its relationship with the rest of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+system.
+XFS contains multiple layers of checking to try to prevent

inconsistent

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata from wreaking havoc on the system.
+Each of these layers contributes information that helps the kernel

to make

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+three decisions about the health of a metadata structure:
+
+- Is a part of this structure obviously corrupt

(``XFS_SCRUB_OFLAG_CORRUPT``) ?
+- Is this structure inconsistent with the rest of the system +  (``XFS_SCRUB_OFLAG_XCORRUPT``) ?
+- Is there so much damage around the filesystem that cross- referencing is not
+  possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
+- Can the structure be optimized to improve performance or reduce the size of
+  metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
+- Does the structure contain data that is not inconsistent but deserves review
+  by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The following sections describe how the metadata scrubbing process

works.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Metadata Buffer Verification
+````````````````````````````
+
+The lowest layer of metadata protection in XFS are the metadata

verifiers built

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+into the buffer cache.
+These functions perform inexpensive internal consistency checking of

the block

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+itself, and answer these questions:
+
+- Does the block belong to this filesystem?
+
+- Does the block belong to the structure that asked for the read?
+  This assumes that metadata blocks only have one owner, which is

always true
+  in XFS.
+
+- Is the type of data stored in the block within a reasonable range of what
+  scrub is expecting?
+
+- Does the physical location of the block match the location it was read from?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Does the block checksum match the data?
+
+The scope of the protections here are very limited -- verifiers can

only
+establish that the filesystem code is reasonably free of gross corruption bugs

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+and that the storage system is reasonably competent at retrieval.
+Corruption problems observed at runtime cause the generation of

health reports,
+failed system calls, and in the extreme case, filesystem shutdowns if the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+corrupt metadata force the cancellation of a dirty transaction.
+
+Every online fsck scrubbing function is expected to read every

ondisk metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+block of a structure in the course of checking the structure.
+Corruption problems observed during a check are immediately reported

to
+userspace as corruption; during a cross-reference, they are reported as a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+failure to cross-reference once the full examination is complete.
+Reads satisfied by a buffer already in cache (and hence already

verified)

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 1080

Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+bypass these checks.
+
+Internal Consistency Checks
+```````````````````````````
+
+The next higher level of metadata protection is the internal record

"After the buffer cache, the next level of metadata protection is..."

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 1088

Quoted 12 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+verification code built into the filesystem.
 
+These checks are split between the buffer verifiers, the in-

filesystem users of
+the buffer cache, and the scrub code itself, depending on the amount of higher

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+level context required.
+The scope of checking is still internal to the block.
+For performance reasons, regular code may skip some of these checks

unless

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+debugging is enabled or a write is about to occur.
+Scrub functions, of course, must check all possible problems.

I'd put this chunk after the list below.

Quoted 2 lines

+Either way, these higher level checking functions answer these questions:

Then this becomes:
"These higher level checking functions..."

Quoted 62 lines

+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the read?
+
+- If the block contains records, do the records fit within the block?
+
+- If the block tracks internal free space information, is it consistent with
+  the record areas?
+
+- Are the records contained inside the block free of obvious corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Record checks in this category are more rigorous and more time-

intensive.
+For example, block pointers and inumbers are checked to ensure that they point
+within the dynamically allocated parts of an allocation group and within

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem.
+Names are checked for invalid characters, and flags are checked for

invalid

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+combinations.
+Other record attributes are checked for sensible values.
+Btree records spanning an interval of the btree keyspace are checked

for
+correct order and lack of mergeability (except for file fork mappings).

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Validation of Userspace-Controlled Record Attributes
+````````````````````````````````````````````````````
+
+Various pieces of filesystem metadata are directly controlled by

userspace.
+Because of this nature, validation work cannot be more precise than checking

Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+that a value is within the possible range.
+These fields include:
+
+- Superblock fields controlled by mount options
+- Filesystem labels
+- File timestamps
+- File permissions
+- File size
+- File flags
+- Names present in directory entries, extended attribute keys, and

filesystem
+  labels

+- Extended attribute key namespaces
+- Extended attribute values
+- File data block contents
+- Quota limits
+- Quota timer expiration (if resource usage exceeds the soft limit)
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Cross-Referencing Space Metadata
+````````````````````````````````
+
+The next higher level of checking is cross-referencing records
Quoted 1 line

between metadata

I kinda like the list first so that the reader has an idea of what these checks are before getting into discussion about them. It just makes it a little more obvious as to why it's "prohibitively expensive" or "dependent on the context of the structure" after having just looked at it

The rest looks good from here.

Allison

Quoted 421 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+structures.
+For regular runtime code, the cost of these checks is considered to

be

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+prohibitively expensive, but as scrub is dedicated to rooting out
+inconsistencies, it must pursue all avenues of inquiry.
+The exact set of cross-referencing is highly dependent on the

context of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+data structure being checked.
+
+The XFS btree code has keyspace scanning functions that online fsck

uses to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+cross reference one structure with another.
+Specifically, scrub can scan the key space of an index to determine

if that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+keyspace is fully, sparsely, or not at all mapped to records.
+For the reverse mapping btree, it is possible to mask parts of the

key for the
+purposes of performing a keyspace scan so that scrub can decide if the rmap
+btree contains records mapping a certain extent of physical space without the

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+sparsenses of the rest of the rmap keyspace getting in the way.
+
+Btree blocks undergo the following checks before cross-referencing:
+
+- Does the type of data stored in the block match what scrub is

expecting?
+
+- Does the block belong to the owning structure that asked for the read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the btree point to valid block addresses

for the type
+  of btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each node block record, does the record key accurate reflect

the contents
+  of the child block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Space allocation records are cross-referenced as follows:
+
+1. Any space mentioned by any metadata structure are cross-

referenced as
+   follows:
+
+   - Does the reverse mapping index list only the appropriate owner as the
+     owner of each block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+   - Are none of the blocks claimed as free space?
+
+   - If these aren't file data blocks, are none of the blocks

claimed as space
+     shared by different owners?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+2. Btree blocks are cross-referenced as follows:
+
+   - Everything in class 1 above.
+
+   - If there's a parent node block, do the keys listed for this

block match the
+     keyspace of this block?
+
+   - Do the sibling pointers point to valid blocks?  Of the same level?
+
+   - Do the child pointers point to valid blocks?  Of the next level down?

Documentation/filesystems/xfs-online-fsck-design.rst +22 -0
+
+3. Free space btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Does the reverse mapping index list no owners of this space?
+
+   - Is this space not claimed by the inode index for inodes?
+
+   - Is it not mentioned by the reference count index?
+
+   - Is there a matching record in the other free space btree?
+
+4. Inode btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is there a matching record in free inode btree?
+
+   - Do cleared bits in the holemask correspond with inode clusters?
+
+   - Do set bits in the freemask correspond with inode records with

zero link
+     count?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+5. Inode records are cross-referenced as follows:
+
+   - Everything in class 1.
+
+   - Do all the fields that summarize information about the file

forks actually
+     match those forks?
+
+   - Does each inode with zero link count correspond to a record in the free
+     inode btree?

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+6. File fork space mapping records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is this space not mentioned by the inode btrees?
+
+   - If this is a CoW fork mapping, does it correspond to a CoW

entry in the
+     reference count btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+7. Reference count records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Within the space subkeyspace of the rmap btree (that is to say,

all
+     records mapped to a particular space extent and ignoring the owner info),
+     are there the same number of reverse mapping records for each block as the
+     reference count record claims?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets are the series to find gaps in
+`refcount btree
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-refcount-gaps>`_,
+`inode btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-inobt-gaps>`_, and
+`rmap btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-rmapbt-gaps>`_ records;
+to find
+`mergeable records
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-mergeable-records>`_;
+and to
+`improve cross referencing with rmap
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-strengthen-rmap-checking>`_

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+before starting a repair.
+
+Checking Extended Attributes
+````````````````````````````
+
+Extended attributes implement a key-value store that enable

fragments of data

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+to be attached to any file.
+Both the kernel and userspace can access the keys and values,

subject to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+namespace and privilege restrictions.
+Most typically these fragments are metadata about the file --

origins, security

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+contexts, user-supplied labels, indexing information, etc.
+
+Names can be as long as 255 bytes and can exist in several different
+namespaces.
+Values can be as large as 64KB.
+A file's extended attributes are stored in blocks mapped by the attr

fork.
+The mappings point to leaf blocks, remote value blocks, or dabtree blocks.
+Block 0 in the attribute fork is always the top of the structure, but otherwise
+each of the three types of blocks can be found at any offset in the attr fork.
+Leaf blocks contain attribute key records that point to the name and the value.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Names are always stored elsewhere in the same leaf block.
+Values that are less than 3/4 the size of a filesystem block are

also stored

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+elsewhere in the same leaf block.
+Remote value blocks contain values that are too large to fit inside

a leaf.
+If the leaf information exceeds a single filesystem block, a dabtree (also
+rooted at block 0) is created to map hashes of the attribute names to leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the attr fork.
+
+Checking an extended attribute structure is not so straightfoward

due to the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+lack of separation between attr blocks and index blocks.
+Scrub must read each block mapped by the attr fork and ignore the

non-leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks:
+
+1. Walk the dabtree in the attr fork (if present) to ensure that

there are no
+   irregularities in the blocks or dabtree mappings that do not point to
+   attr leaf blocks.

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+2. Walk the blocks of the attr fork looking for leaf blocks.
+   For each entry inside a leaf:
+
+   a. Validate that the name does not contain invalid characters.
+
+   b. Read the attr value.
+      This performs a named lookup of the attr name to ensure the

correctness
+      of the dabtree.
+      If the value is stored in a remote block, this also validates the
+      integrity of the remote value block.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing Directories
+``````````````````````````````````````````
+
+The filesystem directory tree is a directed acylic graph structure,

with files
+constituting the nodes, and directory entries (dirents) constituting the edges.
+Directories are a special type of file containing a set of mappings from a
+255-byte sequence (name) to an inumber.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+These are called directory entries, or dirents for short.
+Each directory file must have exactly one directory pointing to the

file.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+A root directory points to itself.
+Directory entries point to files of any type.
+Each non-directory file may have multiple directories point to it.
+
+In XFS, directories are implemented as a file containing up to three

32GB

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+partitions.
+The first partition contains directory entry data blocks.
+Each data block contains variable-sized records associating a user-

provided

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+name with an inumber and, optionally, a file type.
+If the directory entry data grows beyond one block, the second

partition (which
+exists as post-EOF extents) is populated with a block containing free space
+information and an index that maps hashes of the dirent names to directory data

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the first partition.
+This makes directory name lookups very fast.
+If this second partition grows beyond one block, the third partition

is

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+populated with a linear array of free space information for faster
+expansions.
+If the free space has been separated and the second partition grows

again
+beyond one block, then a dabtree is used to map hashes of dirent names to

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+directory data blocks.
+
+Checking a directory is pretty straightfoward:
+
+1. Walk the dabtree in the second partition (if present) to ensure

that there
+   are no irregularities in the blocks or dabtree mappings that do not point to
+   dirent blocks.
+
+2. Walk the blocks of the first partition looking for directory entries.
+   Each dirent is checked as follows:

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+   a. Does the name contain no invalid characters?
+
+   b. Does the inumber correspond to an actual, allocated inode?
+
+   c. Does the child inode have a nonzero link count?
+
+   d. If a file type is included in the dirent, does it match the

type of the
+      inode?
+
+   e. If the child is a subdirectory, does the child's dotdot pointer point
+      back to the parent?
+
+   f. If the directory has a second partition, perform a named lookup of the
+      dirent name to ensure the correctness of the dabtree. +
+3. Walk the free space list in the third partition (if present) to ensure that
+   the free spaces it describes are really unused.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Checking operations involving :ref:`parents <dirparent>` and
+:ref:`file link counts <nlinks>` are discussed in more detail in

later

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+sections.
+
+Checking Directory/Attribute Btrees
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As stated in previous sections, the directory/attribute btree

(dabtree) index
+maps user-provided names to improve lookup times by avoiding linear scans.
+Internally, it maps a 32-bit hash of the name to a block offset within the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate file fork.
+
+The internal structure of a dabtree closely resembles the btrees

that record
+fixed-size metadata records -- each dabtree block contains a magic number, a
+checksum, sibling pointers, a UUID, a tree level, and a log sequence number.
+The format of leaf and node records are the same -- each entry points to the
+next level down in the hierarchy, with dabtree node records pointing to dabtree
+leaf blocks, and dabtree leaf records pointing to non-dabtree blocks elsewhere

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+in the fork.
+
+Checking and cross-referencing the dabtree is very similar to what

is done for

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+space btrees:
+
+- Does the type of data stored in the block match what scrub is

expecting?
+
+- Does the block belong to the owning structure that asked for the read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the dabtree point to valid fork offsets

for dabtree
+  blocks?
+
+- Do leaf pointers within the dabtree point to valid fork offsets for directory
+  or attr leaf blocks?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each dabtree node record, does the record key accurate reflect

the
+  contents of the child dabtree block?
+
+- For each dabtree leaf record, does the record key accurate reflect the
+  contents of the directory or attr block?

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Cross-Referencing Summary Counters
+``````````````````````````````````
+
+XFS maintains three classes of summary counters: available

resources, quota

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+resource usage, and file link counts.
+
+In theory, the amount of available resources (data blocks, inodes,

realtime

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+extents) can be found by walking the entire filesystem.
+This would make for very slow reporting, so a transactional

filesystem can

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+maintain summaries of this information in the superblock.
+Cross-referencing these values against the filesystem metadata

should be a
+simple matter of walking the free space and inode metadata in each AG and the
+realtime bitmap, but there are complications that will be discussed in
+:ref:more detail <fscounters> later.

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+:ref:`Quota usage <quotacheck>` and :ref:`file link count <nlinks>`
+checking are sufficiently complicated to warrant separate sections.
+
+Post-Repair Reverification
+``````````````````````````
+
+After performing a repair, the checking code is run a second time to

validate
+the new structure, and the results of the health assessment are recorded

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+internally and returned to the calling process.
+This step is critical for enabling system administrator to monitor

the status

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the filesystem and the progress of any repairs.
+For developers, it is a useful means to judge the efficacy of error

detection

Documentation/filesystems/xfs-online-fsck-design.rst +1 -0
+and correction in the online and offline checking tools.
(quoted diff) +0 -0

metadata.rst b/Documentation/filesystems/xfs-self-describing- metadata.rst
index b79dbf36dc94..a10c4ae6955e 100644

Documentation/filesystems/xfs-self-describing-metadata.rst +0 -0
@@ -1,4 +1,5 @@

 .. SPDX-License-Identifier: GPL-2.0
+.. _xfs_self_describing_metadata:

 ============================
 XFS Self Describing Metadata

#22 lore raw

Re: [PATCH 05/14] xfs: document the filesystem metadata checking strategy

Quoted 70 lines · Allison Henderson
Quoted 63 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Begin the fifth chapter of the online fsck design documentation, where
we discuss the details of the data structures and algorithms used by the
kernel to examine filesystem metadata and cross-reference it around the
filesystem.

---
 .../filesystems/xfs-online-fsck-design.rst         |  579
++++++++++++++++++++
 .../filesystems/xfs-self-describing-metadata.rst   |    1

 2 files changed, 580 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 42e82971e036..f45bf97fa9c4 100644
@@ -864,3 +864,582 @@ Proposed patchsets include

 and
 `preservation of sickness info during memory reclaim <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=indirect-health-reporting>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+5. Kernel Algorithms and Data Structures
+========================================
+
+This section discusses the key algorithms and data structures of the

kernel
+code that provide the ability to check and repair metadata while the system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+is running.
+The first chapters in this section reveal the pieces that provide

the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+foundation for checking metadata.
+The remainder of this section presents the mechanisms through which

XFS

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+regenerates itself.
+
+Self Describing Metadata
+------------------------
+
+Starting with XFS version 5 in 2012, XFS updated the format of

nearly every
+ondisk block header to record a magic number, a checksum, a universally
+"unique" identifier (UUID), an owner code, the ondisk address of the block,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+and a log sequence number.
+When loading a block buffer from disk, the magic number, UUID,

owner, and
+ondisk address confirm that the retrieved block matches the specific owner of
+the current filesystem, and that the information contained in the block is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+supposed to be found at the ondisk address.
+The first three components enable checking tools to disregard
Quoted 4 lines

alleged metadata
+that doesn't belong to the filesystem, and the fourth component enables the
+filesystem to detect lost writes.

Add...

Quoted 4 lines

"When ever a file system operation modifies a block, the change is submitted to the journal as a transaction. The journal then processes these transactions marking them done once they are safely committed to the disk"

Ok, I'll add that transition. Though I'll s/journal/log/ since this is xfs. :)

Quoted 4 lines

At this point we havnt talked much at all about transactions or logs, and we've just barely begin to cover blocks. I think you at least want a quick blip to describe the relation of these two things, or it may not be clear why we suddenly jumped into logs.

Point taken. Thanks for the suggestion.

Quoted 7 lines

Documentation/filesystems/xfs-online-fsck-design.rst

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The logging code maintains the checksum and the log sequence number

of the last

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+transactional update.
+Checksums are useful for detecting torn writes and other mischief

"Checksums (or crc's) are useful for detecting incomplete or torn writes as well as other discrepancies..."

Checksums are a general concept, whereas CRCs denote a particular family of checksums. The statement would still apply even if we used a different family (e.g. erasure codes, cryptographic hash functions) of function instead of crc32c.

I will, however, avoid the undefined term 'mischief'. Thanks for the correction.

"Checksums are useful for detecting torn writes and other discrepancies that can be introduced between the computer and its storage devices."

Quoted 37 lines
Quoted 30 lines

between the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer and its storage devices.
+Sequence number tracking enables log recovery to avoid applying out

of date

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+log updates to the filesystem.
+
+These two features improve overall runtime resiliency by providing a

means for
+the filesystem to detect obvious corruption when reading metadata blocks from
+disk, but these buffer verifiers cannot provide any consistency checking

Documentation/filesystems/xfs-online-fsck-design.rst +9 -0
+between metadata structures.
+
+For more information, please see the documentation for
+Documentation/filesystems/xfs-self-describing-metadata.rst
+
+Reverse Mapping
+---------------
+
+The original design of XFS (circa 1993) is an improvement upon 1980s

Unix

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem design.
+In those days, storage density was expensive, CPU time was scarce,

and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+excessive seek time could kill performance.
+For performance reasons, filesystem authors were reluctant to add

redundancy to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem, even at the cost of data integrity.
+Filesystems designers in the early 21st century choose different
Quoted 4 lines

strategies to
+increase internal redundancy -- either storing nearly identical copies of
+metadata, or more space-efficient techniques such as erasure coding.

"such as erasure coding which may encode sections of the data with redundant symbols and in more than one location"

Quoted 4 lines

That ties it into the next line. If you go on to talk about a term you have not previously defined, i think you want to either define it quickly or just drop it all together. Right now your goal is to just give the reader context, so you want it to move quickly.

How about I shorten it to:

"...or more space-efficient encoding techniques." ?

and end the paragraph there?

Quoted 19 lines

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 923

Quoted 3 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+Obvious corruptions are typically repaired by copying replicas or
+reconstructing from codes.
+

I think I would have just jumped straight from xfs history to modern xfs...

Quoted 13 lines

+For XFS, a different redundancy strategy was chosen to modernize the design:
+a secondary space usage index that maps allocated disk extents back to their

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+owners.
+By adding a new index, the filesystem retains most of its ability to

scale
+well to heavily threaded workloads involving large datasets, since the primary
+file metadata (the directory tree, the file block map, and the allocation
+groups) remain unchanged.

Quoted 6 lines
Quoted 4 lines

+Although the reverse-mapping feature increases overhead costs for space
+mapping activities just like any other system that improves redundancy, it

"Like any system that improves redundancy, the reverse-mapping feature increases overhead costs for space mapping activities. However, it..."

I like this better. These two sentences have been changed to read:

"Like any system that improves redundancy, the reverse-mapping feature increases overhead costs for space mapping activities. However, it has two critical advantages: first, the reverse index is key to enabling online fsck and other requested functionality such as free space defragmentation, better media failure reporting, and filesystem shrinking."

Quoted 35 lines
Quoted 28 lines

+has two critical advantages: first, the reverse index is key to enabling online
+fsck and other requested functionality such as filesystem reorganization,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+better media failure reporting, and shrinking.
+Second, the different ondisk storage format of the reverse mapping

btree
+defeats device-level deduplication, because the filesystem requires real

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+redundancy.
+
+A criticism of adding the secondary index is that it does nothing to

improve

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the robustness of user data storage itself.
+This is a valid point, but adding a new index for file data block

checksums
+increases write amplification and turns data overwrites into copy- writes, which

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+age the filesystem prematurely.
+In keeping with thirty years of precedent, users who want file data

integrity

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+can supply as powerful a solution as they require.
+As for metadata, the complexity of adding a new secondary index of

space usage
+is much less than adding volume management and storage device mirroring to XFS

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+itself.
+Perfection of RAID and volume management are best left to existing
Quoted 2 lines

layers in
+the kernel.

I think I would cull the entire above paragraph. rmap, crc and raid all have very different points of redundancy, so criticism that an apple is not an orange or visavis just feels like a shortsighted comparison that's probably more of a distraction than anything.

Quoted 3 lines

Sometimes it feels like this document kinda gets off into tangents like it's preemptively trying to position it's self for an argument that hasn't happened yet.

It does! Each of the many tangents that you've pointed out are a reaction to some discussion that we've had on the list, or at an LSF, or <cough> fs nerds sniping on social media. The reason I capture all of these offtopic arguments is to discourage people from wasting time rehashing discussions that were settled long ago.

Admittedly, that is a very defensive reaction on my part...

Quoted 5 lines

But I think it has the effect of pulling the
readers attention off topic into an argument they never thought to consider in the first place. The topic of this section is to explain what rmap is. So lets stay on topic and finish laying out that ground work first before getting into how it compares to other solutions

...and you're right to point out that mentioning these things is distracting and provides fuel to reignite a flamewar. At the same time, I think there's value in identifying the roads not taken, and why.

What if I turned these tangents into explicitly labelled sidebars? Would that help readers who want to stick to the topic?

Quoted 47 lines
Quoted 47 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The information captured in a reverse space mapping record is as

follows:

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+.. code-block:: c
+
+       struct xfs_rmap_irec {
+           xfs_agblock_t    rm_startblock;   /* extent start block

*/
+           xfs_extlen_t     rm_blockcount;   /* extent length */ +           uint64_t         rm_owner;        /* extent owner */ +           uint64_t         rm_offset;       /* offset within the owner */
+           unsigned int     rm_flags;        /* state flags */ +       };

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The first two fields capture the location and size of the physical

space,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in units of filesystem blocks.
+The owner field tells scrub which metadata structure or file inode

have been

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+assigned this space.
+For space allocated to files, the offset field tells scrub where the

space was

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+mapped within the file fork.
+Finally, the flags field provides extra information about the space

usage --
+is this an attribute fork extent?  A file mapping btree extent?  Or an

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+unwritten data extent?
+
+Online filesystem checking judges the consistency of each primary

metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+record by comparing its information against all other space indices.
+The reverse mapping index plays a key role in the consistency

checking process
+because it contains a centralized alternate copy of all space allocation

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+information.
+Program runtime and ease of resource acquisition are the only real

limits to

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+what online checking can consult.
+For example, a file data extent mapping can be checked against:
+
+* The absence of an entry in the free space information.
+* The absence of an entry in the inode index.
+* The absence of an entry in the reference count data if the file is
Quoted 8 lines
Quoted 7 lines

not
+  marked as having shared extents.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+* The correspondence of an entry in the reverse mapping information.
+
+A key observation here is that only the reverse mapping can provide

a positive
+affirmation of correctness if the primary metadata is in doubt.

if any of the above metadata is in doubt...

Fixed.

Quoted 15 lines
Quoted 9 lines

+The checking code for most primary metadata follows a path similar to the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+one outlined above.
+
+A second observation to make about this secondary index is that

proving its

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency with the primary metadata is difficult.
 
+Demonstrating that a given reverse mapping record exactly
Quoted 4 lines

corresponds to the
+primary space metadata involves a full scan of all primary space metadata,
+which is very time intensive.

"But why?" Wonders the reader. Just jump into an example:

Quoted 3 lines

"In order to verify that an rmap extent does not incorrectly over lap with another record, we would need a full scan of all the other records, which is time intensive."

I want to shorten it even further:

"Validating that reverse mapping records are correct requires a full scan of all primary space metadata, which is very time intensive."

Quoted 4 lines

?

And then the below is a separate observation right?

Right.

Quoted 7 lines
Quoted 5 lines

+Scanning activity for online fsck can only use non-blocking lock acquisition
+primitives if the locking order is not the regular order as used by the rest of
+the filesystem.

Lastly, it should be noted that most file system operations tend to lock primary metadata before locking the secondary metadata.

This isn't accurate -- metadata structures don't have separate locks. So it's not true to say that we lock primary or secondary metadata.

We /can/ say that file operations lock the inode, then the AGI, then the AGF; or that directory operations lock the parent and child ILOCKs in inumber order; and that if scrub wants to take locks in any other order, it can only do that via trylocks and backoff.

Quoted 16 lines

This
means that scanning operations that acquire the secondary metadata first may need to yield the secondary lock to filesystem operations that have already acquired the primary lock.

?

Quoted 7 lines

+This means that forward progress during this part of a scan of the reverse
+mapping data cannot be guaranteed if system load is especially heavy.
+Therefore, it is not practical for online check to detect reverse mapping
+records that lack a counterpart in the primary metadata.

Such as <quick list / quick example>

Quoted 3 lines
Quoted 3 lines

+Instead, scrub relies on rigorous cross-referencing during the primary space
+mapping structure checks.

I've converted this section into a bullet list:

"There are several observations to make about reverse mapping indices:

"1. Reverse mappings can provide a positive affirmation of correctness if any of the above primary metadata are in doubt. The checking code for most primary metadata follows a path similar to the one outlined above.

"2. Proving the consistency of secondary metadata with the primary metadata is difficult because that requires a full scan of all primary space metadata, which is very time intensive. For example, checking a reverse mapping record for a file extent mapping btree block requires locking the file and searching the entire btree to confirm the block. Instead, scrub relies on rigorous cross-referencing during the primary space mapping structure checks.

"3. Consistency scans must use non-blocking lock acquisition primitives if the required locking order is not the same order used by regular filesystem operations. This means that forward progress during this part of a scan of the reverse mapping data cannot be guaranteed if system load is heavy."

Quoted 5 lines
Quoted 1 line

+

The below paragraph sounds like a re-cap?

"So to recap, reverse mappings also...."

Yep.

Quoted 7 lines
Quoted 7 lines

+Reverse mappings also play a key role in reconstruction of primary metadata.
+The secondary information is general enough for online repair to synthesize a
+complete copy of any primary space management metadata by locking that
+resource, querying all reverse mapping indices looking for records

Quoted 8 lines
Quoted 6 lines

matching
+the relevant resource, and transforming the mapping into an appropriate format.
+The details of how these records are staged, written to disk, and committed
+into the filesystem are covered in subsequent sections.

I also think the section would be ok if you were to trim off this last paragraph too.

Hm. I still want to set up the expectation that there's more to come. How about a brief two-sentence transition paragraph:

"In summary, reverse mappings play a key role in reconstruction of primary metadata. The details of how these records are staged, written to disk, and committed into the filesystem are covered in subsequent sections."

Quoted 80 lines
Quoted 80 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing
+------------------------------
+
+The first step of checking a metadata structure is to examine every

record
+contained within the structure and its relationship with the rest of the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+system.
+XFS contains multiple layers of checking to try to prevent

inconsistent

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata from wreaking havoc on the system.
+Each of these layers contributes information that helps the kernel

to make

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+three decisions about the health of a metadata structure:
+
+- Is a part of this structure obviously corrupt

(``XFS_SCRUB_OFLAG_CORRUPT``) ?
+- Is this structure inconsistent with the rest of the system +  (``XFS_SCRUB_OFLAG_XCORRUPT``) ?
+- Is there so much damage around the filesystem that cross- referencing is not
+  possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
+- Can the structure be optimized to improve performance or reduce the size of
+  metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
+- Does the structure contain data that is not inconsistent but deserves review
+  by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The following sections describe how the metadata scrubbing process

works.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Metadata Buffer Verification
+````````````````````````````
+
+The lowest layer of metadata protection in XFS are the metadata

verifiers built

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+into the buffer cache.
+These functions perform inexpensive internal consistency checking of

the block

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+itself, and answer these questions:
+
+- Does the block belong to this filesystem?
+
+- Does the block belong to the structure that asked for the read?
+  This assumes that metadata blocks only have one owner, which is

always true
+  in XFS.
+
+- Is the type of data stored in the block within a reasonable range of what
+  scrub is expecting?
+
+- Does the physical location of the block match the location it was read from?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Does the block checksum match the data?
+
+The scope of the protections here are very limited -- verifiers can

only
+establish that the filesystem code is reasonably free of gross corruption bugs

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+and that the storage system is reasonably competent at retrieval.
+Corruption problems observed at runtime cause the generation of

health reports,
+failed system calls, and in the extreme case, filesystem shutdowns if the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+corrupt metadata force the cancellation of a dirty transaction.
+
+Every online fsck scrubbing function is expected to read every

ondisk metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+block of a structure in the course of checking the structure.
+Corruption problems observed during a check are immediately reported

to
+userspace as corruption; during a cross-reference, they are reported as a

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+failure to cross-reference once the full examination is complete.
+Reads satisfied by a buffer already in cache (and hence already
Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 1080

Quoted 7 lines

verified)

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+bypass these checks.
+
+Internal Consistency Checks
+```````````````````````````
+
+The next higher level of metadata protection is the internal record

"After the buffer cache, the next level of metadata protection is..."

Changed. I'll do the same to the next section as well.

Quoted 14 lines

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 1088

Quoted 12 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+verification code built into the filesystem.
 
+These checks are split between the buffer verifiers, the in-

filesystem users of
+the buffer cache, and the scrub code itself, depending on the amount of higher

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+level context required.
+The scope of checking is still internal to the block.
+For performance reasons, regular code may skip some of these checks

unless

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+debugging is enabled or a write is about to occur.
+Scrub functions, of course, must check all possible problems.

I'd put this chunk after the list below.

Quoted 4 lines
Quoted 2 lines

+Either way, these higher level checking functions answer these questions:

Then this becomes:
"These higher level checking functions..."

Done.

Quoted 61 lines
Quoted 61 lines

+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the read?
+
+- If the block contains records, do the records fit within the block?
+
+- If the block tracks internal free space information, is it consistent with
+  the record areas?
+
+- Are the records contained inside the block free of obvious corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Record checks in this category are more rigorous and more time-

intensive.
+For example, block pointers and inumbers are checked to ensure that they point
+within the dynamically allocated parts of an allocation group and within

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem.
+Names are checked for invalid characters, and flags are checked for

invalid

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+combinations.
+Other record attributes are checked for sensible values.
+Btree records spanning an interval of the btree keyspace are checked

for
+correct order and lack of mergeability (except for file fork mappings).

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Validation of Userspace-Controlled Record Attributes
+````````````````````````````````````````````````````
+
+Various pieces of filesystem metadata are directly controlled by

userspace.
+Because of this nature, validation work cannot be more precise than checking

Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+that a value is within the possible range.
+These fields include:
+
+- Superblock fields controlled by mount options
+- Filesystem labels
+- File timestamps
+- File permissions
+- File size
+- File flags
+- Names present in directory entries, extended attribute keys, and

filesystem
+  labels

+- Extended attribute key namespaces
+- Extended attribute values
+- File data block contents
+- Quota limits
+- Quota timer expiration (if resource usage exceeds the soft limit)
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Cross-Referencing Space Metadata
+````````````````````````````````
+
Quoted 8 lines
Quoted 2 lines

+The next higher level of checking is cross-referencing records between metadata

I kinda like the list first so that the reader has an idea of what these checks are before getting into discussion about them. It just makes it a little more obvious as to why it's "prohibitively expensive" or "dependent on the context of the structure" after having just looked at it

<nod>

Quoted 1 line

The rest looks good from here.

Woot. Onto the next reply! :)

--D

Quoted 424 lines

Allison

Quoted 421 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+structures.
+For regular runtime code, the cost of these checks is considered to

be

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+prohibitively expensive, but as scrub is dedicated to rooting out
+inconsistencies, it must pursue all avenues of inquiry.
+The exact set of cross-referencing is highly dependent on the

context of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+data structure being checked.
+
+The XFS btree code has keyspace scanning functions that online fsck

uses to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+cross reference one structure with another.
+Specifically, scrub can scan the key space of an index to determine

if that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+keyspace is fully, sparsely, or not at all mapped to records.
+For the reverse mapping btree, it is possible to mask parts of the

key for the
+purposes of performing a keyspace scan so that scrub can decide if the rmap
+btree contains records mapping a certain extent of physical space without the

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+sparsenses of the rest of the rmap keyspace getting in the way.
+
+Btree blocks undergo the following checks before cross-referencing:
+
+- Does the type of data stored in the block match what scrub is

expecting?
+
+- Does the block belong to the owning structure that asked for the read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the btree point to valid block addresses

for the type
+  of btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each node block record, does the record key accurate reflect

the contents
+  of the child block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Space allocation records are cross-referenced as follows:
+
+1. Any space mentioned by any metadata structure are cross-

referenced as
+   follows:
+
+   - Does the reverse mapping index list only the appropriate owner as the
+     owner of each block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+   - Are none of the blocks claimed as free space?
+
+   - If these aren't file data blocks, are none of the blocks

claimed as space
+     shared by different owners?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+2. Btree blocks are cross-referenced as follows:
+
+   - Everything in class 1 above.
+
+   - If there's a parent node block, do the keys listed for this

block match the
+     keyspace of this block?
+
+   - Do the sibling pointers point to valid blocks?  Of the same level?
+
+   - Do the child pointers point to valid blocks?  Of the next level down?

Documentation/filesystems/xfs-online-fsck-design.rst +22 -0
+
+3. Free space btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Does the reverse mapping index list no owners of this space?
+
+   - Is this space not claimed by the inode index for inodes?
+
+   - Is it not mentioned by the reference count index?
+
+   - Is there a matching record in the other free space btree?
+
+4. Inode btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is there a matching record in free inode btree?
+
+   - Do cleared bits in the holemask correspond with inode clusters?
+
+   - Do set bits in the freemask correspond with inode records with

zero link
+     count?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+5. Inode records are cross-referenced as follows:
+
+   - Everything in class 1.
+
+   - Do all the fields that summarize information about the file

forks actually
+     match those forks?
+
+   - Does each inode with zero link count correspond to a record in the free
+     inode btree?

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+6. File fork space mapping records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is this space not mentioned by the inode btrees?
+
+   - If this is a CoW fork mapping, does it correspond to a CoW

entry in the
+     reference count btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+7. Reference count records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Within the space subkeyspace of the rmap btree (that is to say,

all
+     records mapped to a particular space extent and ignoring the owner info),
+     are there the same number of reverse mapping records for each block as the
+     reference count record claims?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets are the series to find gaps in
+`refcount btree
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-refcount-gaps>`_,
+`inode btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-inobt-gaps>`_, and
+`rmap btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-rmapbt-gaps>`_ records;
+to find
+`mergeable records
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-mergeable-records>`_;
+and to
+`improve cross referencing with rmap
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-strengthen-rmap-checking>`_

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+before starting a repair.
+
+Checking Extended Attributes
+````````````````````````````
+
+Extended attributes implement a key-value store that enable

fragments of data

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+to be attached to any file.
+Both the kernel and userspace can access the keys and values,

subject to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+namespace and privilege restrictions.
+Most typically these fragments are metadata about the file --

origins, security

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+contexts, user-supplied labels, indexing information, etc.
+
+Names can be as long as 255 bytes and can exist in several different
+namespaces.
+Values can be as large as 64KB.
+A file's extended attributes are stored in blocks mapped by the attr

fork.
+The mappings point to leaf blocks, remote value blocks, or dabtree blocks.
+Block 0 in the attribute fork is always the top of the structure, but otherwise
+each of the three types of blocks can be found at any offset in the attr fork.
+Leaf blocks contain attribute key records that point to the name and the value.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Names are always stored elsewhere in the same leaf block.
+Values that are less than 3/4 the size of a filesystem block are

also stored

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+elsewhere in the same leaf block.
+Remote value blocks contain values that are too large to fit inside

a leaf.
+If the leaf information exceeds a single filesystem block, a dabtree (also
+rooted at block 0) is created to map hashes of the attribute names to leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the attr fork.
+
+Checking an extended attribute structure is not so straightfoward

due to the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+lack of separation between attr blocks and index blocks.
+Scrub must read each block mapped by the attr fork and ignore the

non-leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks:
+
+1. Walk the dabtree in the attr fork (if present) to ensure that

there are no
+   irregularities in the blocks or dabtree mappings that do not point to
+   attr leaf blocks.

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+2. Walk the blocks of the attr fork looking for leaf blocks.
+   For each entry inside a leaf:
+
+   a. Validate that the name does not contain invalid characters.
+
+   b. Read the attr value.
+      This performs a named lookup of the attr name to ensure the

correctness
+      of the dabtree.
+      If the value is stored in a remote block, this also validates the
+      integrity of the remote value block.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing Directories
+``````````````````````````````````````````
+
+The filesystem directory tree is a directed acylic graph structure,

with files
+constituting the nodes, and directory entries (dirents) constituting the edges.
+Directories are a special type of file containing a set of mappings from a
+255-byte sequence (name) to an inumber.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+These are called directory entries, or dirents for short.
+Each directory file must have exactly one directory pointing to the

file.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+A root directory points to itself.
+Directory entries point to files of any type.
+Each non-directory file may have multiple directories point to it.
+
+In XFS, directories are implemented as a file containing up to three

32GB

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+partitions.
+The first partition contains directory entry data blocks.
+Each data block contains variable-sized records associating a user-

provided

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+name with an inumber and, optionally, a file type.
+If the directory entry data grows beyond one block, the second

partition (which
+exists as post-EOF extents) is populated with a block containing free space
+information and an index that maps hashes of the dirent names to directory data

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the first partition.
+This makes directory name lookups very fast.
+If this second partition grows beyond one block, the third partition

is

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+populated with a linear array of free space information for faster
+expansions.
+If the free space has been separated and the second partition grows

again
+beyond one block, then a dabtree is used to map hashes of dirent names to

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+directory data blocks.
+
+Checking a directory is pretty straightfoward:
+
+1. Walk the dabtree in the second partition (if present) to ensure

that there
+   are no irregularities in the blocks or dabtree mappings that do not point to
+   dirent blocks.
+
+2. Walk the blocks of the first partition looking for directory entries.
+   Each dirent is checked as follows:

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+   a. Does the name contain no invalid characters?
+
+   b. Does the inumber correspond to an actual, allocated inode?
+
+   c. Does the child inode have a nonzero link count?
+
+   d. If a file type is included in the dirent, does it match the

type of the
+      inode?
+
+   e. If the child is a subdirectory, does the child's dotdot pointer point
+      back to the parent?
+
+   f. If the directory has a second partition, perform a named lookup of the
+      dirent name to ensure the correctness of the dabtree. +
+3. Walk the free space list in the third partition (if present) to ensure that
+   the free spaces it describes are really unused.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Checking operations involving :ref:`parents <dirparent>` and
+:ref:`file link counts <nlinks>` are discussed in more detail in

later

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+sections.
+
+Checking Directory/Attribute Btrees
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As stated in previous sections, the directory/attribute btree

(dabtree) index
+maps user-provided names to improve lookup times by avoiding linear scans.
+Internally, it maps a 32-bit hash of the name to a block offset within the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate file fork.
+
+The internal structure of a dabtree closely resembles the btrees

that record
+fixed-size metadata records -- each dabtree block contains a magic number, a
+checksum, sibling pointers, a UUID, a tree level, and a log sequence number.
+The format of leaf and node records are the same -- each entry points to the
+next level down in the hierarchy, with dabtree node records pointing to dabtree
+leaf blocks, and dabtree leaf records pointing to non-dabtree blocks elsewhere

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+in the fork.
+
+Checking and cross-referencing the dabtree is very similar to what

is done for

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+space btrees:
+
+- Does the type of data stored in the block match what scrub is

expecting?
+
+- Does the block belong to the owning structure that asked for the read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the dabtree point to valid fork offsets

for dabtree
+  blocks?
+
+- Do leaf pointers within the dabtree point to valid fork offsets for directory
+  or attr leaf blocks?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each dabtree node record, does the record key accurate reflect

the
+  contents of the child dabtree block?
+
+- For each dabtree leaf record, does the record key accurate reflect the
+  contents of the directory or attr block?

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Cross-Referencing Summary Counters
+``````````````````````````````````
+
+XFS maintains three classes of summary counters: available

resources, quota

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+resource usage, and file link counts.
+
+In theory, the amount of available resources (data blocks, inodes,

realtime

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+extents) can be found by walking the entire filesystem.
+This would make for very slow reporting, so a transactional

filesystem can

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+maintain summaries of this information in the superblock.
+Cross-referencing these values against the filesystem metadata

should be a
+simple matter of walking the free space and inode metadata in each AG and the
+realtime bitmap, but there are complications that will be discussed in
+:ref:more detail <fscounters> later.

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+:ref:`Quota usage <quotacheck>` and :ref:`file link count <nlinks>`
+checking are sufficiently complicated to warrant separate sections.
+
+Post-Repair Reverification
+``````````````````````````
+
+After performing a repair, the checking code is run a second time to

validate
+the new structure, and the results of the health assessment are recorded

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+internally and returned to the calling process.
+This step is critical for enabling system administrator to monitor

the status

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the filesystem and the progress of any repairs.
+For developers, it is a useful means to judge the efficacy of error

detection

Documentation/filesystems/xfs-online-fsck-design.rst +1 -0
+and correction in the online and offline checking tools.
(quoted diff) +0 -0

metadata.rst b/Documentation/filesystems/xfs-self-describing- metadata.rst
index b79dbf36dc94..a10c4ae6955e 100644

Documentation/filesystems/xfs-self-describing-metadata.rst +0 -0
@@ -1,4 +1,5 @@

 .. SPDX-License-Identifier: GPL-2.0
+.. _xfs_self_describing_metadata:

 ============================
 XFS Self Describing Metadata

#23 lore raw

Re: [PATCH 05/14] xfs: document the filesystem metadata checking strategy

Quoted 176 lines · Darrick J. Wong
Quoted 79 lines · Allison Henderson
Quoted 72 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Begin the fifth chapter of the online fsck design documentation, where
we discuss the details of the data structures and algorithms used by
the
kernel to examine filesystem metadata and cross-reference it around
the
filesystem.

---
 .../filesystems/xfs-online-fsck-design.rst         |  579
++++++++++++++++++++
 .../filesystems/xfs-self-describing-metadata.rst   |    1

 2 files changed, 580 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index 42e82971e036..f45bf97fa9c4 100644
@@ -864,3 +864,582 @@ Proposed patchsets include

 and
 `preservation of sickness info during memory reclaim <
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=indirect-health-reporting>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+5. Kernel Algorithms and Data Structures
+========================================
+
+This section discusses the key algorithms and data structures of

the
kernel
+code that provide the ability to check and repair metadata while the
system

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+is running.
+The first chapters in this section reveal the pieces that

provide
the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+foundation for checking metadata.
+The remainder of this section presents the mechanisms through

which
XFS

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+regenerates itself.
+
+Self Describing Metadata
+------------------------
+
+Starting with XFS version 5 in 2012, XFS updated the format of

nearly every
+ondisk block header to record a magic number, a checksum, a universally
+"unique" identifier (UUID), an owner code, the ondisk address of the
block,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+and a log sequence number.
+When loading a block buffer from disk, the magic number, UUID,

owner, and
+ondisk address confirm that the retrieved block matches the specific
owner of
+the current filesystem, and that the information contained in the
block is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+supposed to be found at the ondisk address.
+The first three components enable checking tools to disregard
Quoted 4 lines

alleged metadata
+that doesn't belong to the filesystem, and the fourth component enables the
+filesystem to detect lost writes.

Add...

Quoted 6 lines

"When ever a file system operation modifies a block, the change is submitted to the journal as a transaction.  The journal then processes
these transactions marking them done once they are safely committed to
the disk"

Ok, I'll add that transition.  Though I'll s/journal/log/ since this is
xfs. :)

Quoted 7 lines

At this point we havnt talked much at all about transactions or logs,
and we've just barely begin to cover blocks.  I think you at least want
a quick blip to describe the relation of these two things, or it may
not be clear why we suddenly jumped into logs.

Point taken.  Thanks for the suggestion.

Quoted 9 lines
Quoted 7 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The logging code maintains the checksum and the log sequence

number
of the last

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+transactional update.
+Checksums are useful for detecting torn writes and other

mischief

"Checksums (or crc's) are useful for detecting incomplete or torn writes as well as other discrepancies..."

Checksums are a general concept, whereas CRCs denote a particular family
of checksums.  The statement would still apply even if we used a different family (e.g. erasure codes, cryptographic hash functions) of
function instead of crc32c.

I will, however, avoid the undefined term 'mischief'.  Thanks for the correction.

"Checksums are useful for detecting torn writes and other discrepancies
that can be introduced between the computer and its storage devices."

Quoted 44 lines
Quoted 36 lines

between the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer and its storage devices.
+Sequence number tracking enables log recovery to avoid applying

out
of date

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+log updates to the filesystem.
+
+These two features improve overall runtime resiliency by

providing a
means for
+the filesystem to detect obvious corruption when reading metadata
blocks from
+disk, but these buffer verifiers cannot provide any consistency checking

Documentation/filesystems/xfs-online-fsck-design.rst +9 -0
+between metadata structures.
+
+For more information, please see the documentation for
+Documentation/filesystems/xfs-self-describing-metadata.rst
+
+Reverse Mapping
+---------------
+
+The original design of XFS (circa 1993) is an improvement upon

1980s
Unix

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+filesystem design.
+In those days, storage density was expensive, CPU time was

scarce,
and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+excessive seek time could kill performance.
+For performance reasons, filesystem authors were reluctant to

add
redundancy to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem, even at the cost of data integrity.
+Filesystems designers in the early 21st century choose different
Quoted 5 lines

strategies to
+increase internal redundancy -- either storing nearly identical copies of
+metadata, or more space-efficient techniques such as erasure coding.

"such as erasure coding which may encode sections of the data with redundant symbols and in more than one location"

Quoted 6 lines

That ties it into the next line.  If you go on to talk about a term you
have not previously defined, i think you want to either define it quickly or just drop it all together.  Right now your goal is to just
give the reader context, so you want it to move quickly.

Quoted 3 lines

How about I shorten it to:

"...or more space-efficient encoding techniques." ?

Sure, I think that would be fine

Quoted 40 lines

and end the paragraph there?

Quoted 25 lines

Documentation/filesystems/xfs-online-fsck-design.rst

Quoted 4 lines

+Obvious corruptions are typically repaired by copying replicas or

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+reconstructing from codes.
+

I think I would have just jumped straight from xfs history to modern
xfs...

Quoted 17 lines

+For XFS, a different redundancy strategy was chosen to modernize the
design:
+a secondary space usage index that maps allocated disk extents back
to their

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+owners.
+By adding a new index, the filesystem retains most of its

ability to
scale
+well to heavily threaded workloads involving large datasets, since
the primary
+file metadata (the directory tree, the file block map, and the allocation
+groups) remain unchanged.

Quoted 9 lines
Quoted 5 lines

+Although the reverse-mapping feature increases overhead costs for
space
+mapping activities just like any other system that improves redundancy, it

"Like any system that improves redundancy, the reverse-mapping feature
increases overhead costs for space mapping activities. However, it..."

I like this better.  These two sentences have been changed to read:

Quoted 8 lines

"Like any system that improves redundancy, the reverse-mapping feature
increases overhead costs for space mapping activities.  However, it has
two critical advantages: first, the reverse index is key to enabling online fsck and other requested functionality such as free space defragmentation, better media failure reporting, and filesystem shrinking."

Alrighty, sounds good

Quoted 65 lines
Quoted 45 lines
Quoted 36 lines

+has two critical advantages: first, the reverse index is key to enabling online
+fsck and other requested functionality such as filesystem reorganization,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+better media failure reporting, and shrinking.
+Second, the different ondisk storage format of the reverse

mapping
btree
+defeats device-level deduplication, because the filesystem requires
real

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+redundancy.
+
+A criticism of adding the secondary index is that it does

nothing to
improve

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the robustness of user data storage itself.
+This is a valid point, but adding a new index for file data

block
checksums
+increases write amplification and turns data overwrites into copy-
writes, which

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+age the filesystem prematurely.
+In keeping with thirty years of precedent, users who want file

data
integrity

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+can supply as powerful a solution as they require.
+As for metadata, the complexity of adding a new secondary index

of
space usage
+is much less than adding volume management and storage device mirroring to XFS

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+itself.
+Perfection of RAID and volume management are best left to
Quoted 3 lines

existing
layers in
+the kernel.

I think I would cull the entire above paragraph.  rmap, crc and raid
all have very different points of redundancy, so criticism that an apple is not an orange or visavis just feels like a shortsighted comparison that's probably more of a distraction than anything.

Quoted 3 lines

Sometimes it feels like this document kinda gets off into tangents like it's preemptively trying to position it's self for an argument that hasn't happened yet.

It does!  Each of the many tangents that you've pointed out are a reaction to some discussion that we've had on the list, or at an LSF, or <cough> fs nerds sniping on social media.  The reason I capture all of these offtopic arguments is to discourage people from wasting time rehashing discussions that were settled long ago.

Admittedly, that is a very defensive reaction on my part...

Quoted 7 lines

But I think it has the effect of pulling the
readers attention off topic into an argument they never thought to consider in the first place.  The topic of this section is to explain
what rmap is.  So lets stay on topic and finish laying out that ground
work first before getting into how it compares to other solutions

Quoted 8 lines

...and you're right to point out that mentioning these things is distracting and provides fuel to reignite a flamewar.  At the same time,
I think there's value in identifying the roads not taken, and why.

What if I turned these tangents into explicitly labelled sidebars? Would that help readers who want to stick to the topic?

Sure, I think that would be a reasonable compromise

Quoted 92 lines
Quoted 60 lines
Quoted 60 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The information captured in a reverse space mapping record is as

follows:

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+.. code-block:: c
+
+       struct xfs_rmap_irec {
+           xfs_agblock_t    rm_startblock;   /* extent start

block
*/
+           xfs_extlen_t     rm_blockcount;   /* extent length */ +           uint64_t         rm_owner;        /* extent owner */ +           uint64_t         rm_offset;       /* offset within the
owner */
+           unsigned int     rm_flags;        /* state flags */ +       };

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The first two fields capture the location and size of the

physical
space,

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in units of filesystem blocks.
+The owner field tells scrub which metadata structure or file

inode
have been

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+assigned this space.
+For space allocated to files, the offset field tells scrub where

the
space was

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+mapped within the file fork.
+Finally, the flags field provides extra information about the

space
usage --
+is this an attribute fork extent?  A file mapping btree extent? Or
an

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+unwritten data extent?
+
+Online filesystem checking judges the consistency of each

primary
metadata
+record by comparing its information against all other space indices.
+The reverse mapping index plays a key role in the consistency checking process
+because it contains a centralized alternate copy of all space allocation

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+information.
+Program runtime and ease of resource acquisition are the only

real
limits to

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+what online checking can consult.
+For example, a file data extent mapping can be checked against:
+
+* The absence of an entry in the free space information.
+* The absence of an entry in the inode index.
+* The absence of an entry in the reference count data if the

file is
not
+  marked as having shared extents.

Quoted 8 lines
Quoted 7 lines

+* The correspondence of an entry in the reverse mapping information.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+A key observation here is that only the reverse mapping can

provide
a positive
+affirmation of correctness if the primary metadata is in doubt.

if any of the above metadata is in doubt...

Fixed.

Quoted 16 lines
Quoted 10 lines

+The checking code for most primary metadata follows a path similar
to the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+one outlined above.
+
+A second observation to make about this secondary index is that

proving its

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency with the primary metadata is difficult.
 
+Demonstrating that a given reverse mapping record exactly
Quoted 4 lines

corresponds to the
+primary space metadata involves a full scan of all primary space metadata,
+which is very time intensive.

"But why?" Wonders the reader. Just jump into an example:

Quoted 4 lines

"In order to verify that an rmap extent does not incorrectly over lap
with another record, we would need a full scan of all the other records, which is time intensive."

Quoted 4 lines

I want to shorten it even further:

"Validating that reverse mapping records are correct requires a full scan of all primary space metadata, which is very time intensive."

Ok, I think that sounds fine

Quoted 20 lines
Quoted 5 lines

?

And then the below is a separate observation right?

Right.

Quoted 8 lines
Quoted 6 lines

+Scanning activity for online fsck can only use non-blocking lock acquisition
+primitives if the locking order is not the regular order as used by
the rest of
+the filesystem.

Lastly, it should be noted that most file system operations tend to lock primary metadata before locking the secondary metadata.

This isn't accurate -- metadata structures don't have separate locks. So it's not true to say that we lock primary or secondary metadata.

Quoted 6 lines

We /can/ say that file operations lock the inode, then the AGI, then the
AGF; or that directory operations lock the parent and child ILOCKs in inumber order; and that if scrub wants to take locks in any other order,
it can only do that via trylocks and backoff.

I see, ok maybe giving one or both of those examples is clearer then

Quoted 46 lines
Quoted 19 lines

This
means that scanning operations that acquire the secondary metadata first may need to yield the secondary lock to filesystem operations that have already acquired the primary lock.

?

Quoted 9 lines

+This means that forward progress during this part of a scan of the
reverse
+mapping data cannot be guaranteed if system load is especially heavy.
+Therefore, it is not practical for online check to detect reverse
mapping
+records that lack a counterpart in the primary metadata.

Such as <quick list / quick example>

Quoted 3 lines
Quoted 3 lines

+Instead, scrub relies on rigorous cross-referencing during the primary space
+mapping structure checks.

I've converted this section into a bullet list:

"There are several observations to make about reverse mapping indices:

"1. Reverse mappings can provide a positive affirmation of correctness if
any of the above primary metadata are in doubt.  The checking code for
most primary metadata follows a path similar to the one outlined above.

"2. Proving the consistency of secondary metadata with the primary metadata is difficult because that requires a full scan of all primary
space metadata, which is very time intensive.  For example, checking a
reverse mapping record for a file extent mapping btree block requires locking the file and searching the entire btree to confirm the block. Instead, scrub relies on rigorous cross-referencing during the primary
space mapping structure checks.

Quoted 6 lines

"3. Consistency scans must use non-blocking lock acquisition primitives
if the required locking order is not the same order used by regular filesystem operations.  This means that forward progress during this part of a scan of the reverse mapping data cannot be guaranteed if system load is heavy."

Ok, I think that reads cleaner

Quoted 34 lines
Quoted 6 lines
Quoted 2 lines

+

The below paragraph sounds like a re-cap?

"So to recap, reverse mappings also...."

Yep.

Quoted 12 lines
Quoted 12 lines

+Reverse mappings also play a key role in reconstruction of primary
metadata.
+The secondary information is general enough for online repair to synthesize a
+complete copy of any primary space management metadata by locking
that
+resource, querying all reverse mapping indices looking for records
matching
+the relevant resource, and transforming the mapping into an

Quoted 8 lines
Quoted 5 lines

appropriate format.
+The details of how these records are staged, written to disk, and
committed
+into the filesystem are covered in subsequent sections.

I also think the section would be ok if you were to trim off this last
paragraph too.

Hm.  I still want to set up the expectation that there's more to come.
How about a brief two-sentence transition paragraph:

Quoted 5 lines

"In summary, reverse mappings play a key role in reconstruction of primary metadata.  The details of how these records are staged, written
to disk, and committed into the filesystem are covered in subsequent sections."

Ok, I think that's a cleaner wrap up

Quoted 714 lines
Quoted 99 lines
Quoted 99 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing
+------------------------------
+
+The first step of checking a metadata structure is to examine

every
record
+contained within the structure and its relationship with the rest of
the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+system.
+XFS contains multiple layers of checking to try to prevent

inconsistent

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+metadata from wreaking havoc on the system.
+Each of these layers contributes information that helps the

kernel
to make

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+three decisions about the health of a metadata structure:
+
+- Is a part of this structure obviously corrupt

(``XFS_SCRUB_OFLAG_CORRUPT``) ?
+- Is this structure inconsistent with the rest of the system +  (``XFS_SCRUB_OFLAG_XCORRUPT``) ?
+- Is there so much damage around the filesystem that cross- referencing is not
+  possible (``XFS_SCRUB_OFLAG_XFAIL``) ?
+- Can the structure be optimized to improve performance or reduce
the size of
+  metadata (``XFS_SCRUB_OFLAG_PREEN``) ?
+- Does the structure contain data that is not inconsistent but deserves review
+  by the system administrator (``XFS_SCRUB_OFLAG_WARNING``) ?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+The following sections describe how the metadata scrubbing

process
works.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Metadata Buffer Verification
+````````````````````````````
+
+The lowest layer of metadata protection in XFS are the metadata

verifiers built

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+into the buffer cache.
+These functions perform inexpensive internal consistency

checking of
the block

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+itself, and answer these questions:
+
+- Does the block belong to this filesystem?
+
+- Does the block belong to the structure that asked for the

read?
+  This assumes that metadata blocks only have one owner, which is
always true
+  in XFS.
+
+- Is the type of data stored in the block within a reasonable range
of what
+  scrub is expecting?
+
+- Does the physical location of the block match the location it was
read from?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Does the block checksum match the data?
+
+The scope of the protections here are very limited -- verifiers

can
only
+establish that the filesystem code is reasonably free of gross corruption bugs
+and that the storage system is reasonably competent at retrieval.
+Corruption problems observed at runtime cause the generation of health reports,
+failed system calls, and in the extreme case, filesystem shutdowns
if the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+corrupt metadata force the cancellation of a dirty transaction.
+
+Every online fsck scrubbing function is expected to read every

ondisk metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+block of a structure in the course of checking the structure.
+Corruption problems observed during a check are immediately

reported
to
+userspace as corruption; during a cross-reference, they are reported
as a
+failure to cross-reference once the full examination is complete.
+Reads satisfied by a buffer already in cache (and hence already verified)
+bypass these checks.

Quoted 8 lines
Quoted 6 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Internal Consistency Checks
+```````````````````````````
+
+The next higher level of metadata protection is the internal

record

"After the buffer cache, the next level of metadata protection is..."

Changed.  I'll do the same to the next section as well.

Quoted 16 lines
Quoted 12 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+verification code built into the filesystem.
 
+These checks are split between the buffer verifiers, the in-

filesystem users of
+the buffer cache, and the scrub code itself, depending on the amount
of higher

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+level context required.
+The scope of checking is still internal to the block.
+For performance reasons, regular code may skip some of these

checks
unless

Documentation/filesystems/xfs-online-fsck-design.rst Proposed patchsets include · line 1088

Quoted 2 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+debugging is enabled or a write is about to occur.
+Scrub functions, of course, must check all possible problems.

I'd put this chunk after the list below.

Quoted 4 lines
Quoted 2 lines

+Either way, these higher level checking functions answer these questions:

Then this becomes:
"These higher level checking functions..."

Done.

Quoted 72 lines
Quoted 70 lines

+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the
read?
+
+- If the block contains records, do the records fit within the block?
+
+- If the block tracks internal free space information, is it consistent with
+  the record areas?
+
+- Are the records contained inside the block free of obvious corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Record checks in this category are more rigorous and more time-

intensive.
+For example, block pointers and inumbers are checked to ensure that
they point
+within the dynamically allocated parts of an allocation group and
within

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+the filesystem.
+Names are checked for invalid characters, and flags are checked

for
invalid

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+combinations.
+Other record attributes are checked for sensible values.
+Btree records spanning an interval of the btree keyspace are

checked
for
+correct order and lack of mergeability (except for file fork mappings).

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Validation of Userspace-Controlled Record Attributes
+````````````````````````````````````````````````````
+
+Various pieces of filesystem metadata are directly controlled by

userspace.
+Because of this nature, validation work cannot be more precise than
checking

Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+that a value is within the possible range.
+These fields include:
+
+- Superblock fields controlled by mount options
+- Filesystem labels
+- File timestamps
+- File permissions
+- File size
+- File flags
+- Names present in directory entries, extended attribute keys,

and
filesystem
+  labels

+- Extended attribute key namespaces
+- Extended attribute values
+- File data block contents
+- Quota limits
+- Quota timer expiration (if resource usage exceeds the soft

limit)

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Cross-Referencing Space Metadata
+````````````````````````````````
+
+The next higher level of checking is cross-referencing records
Quoted 1 line

between metadata

Quoted 8 lines

I kinda like the list first so that the reader has an idea of what these checks are before getting into discussion about them.  It just
makes it a little more obvious as to why it's "prohibitively expensive"
or "dependent on the context of the structure" after having just looked
at it

<nod>

Quoted 1 line

The rest looks good from here.

Woot.  Onto the next reply! :)

--D

Quoted 492 lines

Allison

Quoted 489 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+structures.
+For regular runtime code, the cost of these checks is considered

to
be
+prohibitively expensive, but as scrub is dedicated to rooting out

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistencies, it must pursue all avenues of inquiry.
+The exact set of cross-referencing is highly dependent on the

context of the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+data structure being checked.
+
+The XFS btree code has keyspace scanning functions that online

fsck
uses to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+cross reference one structure with another.
+Specifically, scrub can scan the key space of an index to

determine
if that

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+keyspace is fully, sparsely, or not at all mapped to records.
+For the reverse mapping btree, it is possible to mask parts of

the
key for the
+purposes of performing a keyspace scan so that scrub can decide if
the rmap
+btree contains records mapping a certain extent of physical space
without the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+sparsenses of the rest of the rmap keyspace getting in the way.
+
+Btree blocks undergo the following checks before cross-

referencing:
+
+- Does the type of data stored in the block match what scrub is expecting?
+
+- Does the block belong to the owning structure that asked for the
read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the btree point to valid block

addresses
for the type
+  of btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each node block record, does the record key accurate

reflect
the contents
+  of the child block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Space allocation records are cross-referenced as follows:
+
+1. Any space mentioned by any metadata structure are cross-

referenced as
+   follows:
+
+   - Does the reverse mapping index list only the appropriate owner
as the
+     owner of each block?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+   - Are none of the blocks claimed as free space?
+
+   - If these aren't file data blocks, are none of the blocks

claimed as space
+     shared by different owners?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+2. Btree blocks are cross-referenced as follows:
+
+   - Everything in class 1 above.
+
+   - If there's a parent node block, do the keys listed for this

block match the
+     keyspace of this block?
+
+   - Do the sibling pointers point to valid blocks?  Of the same level?
+
+   - Do the child pointers point to valid blocks?  Of the next level
down?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+3. Free space btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Does the reverse mapping index list no owners of this

space?

Documentation/filesystems/xfs-online-fsck-design.rst +14 -0
+
+   - Is this space not claimed by the inode index for inodes?
+
+   - Is it not mentioned by the reference count index?
+
+   - Is there a matching record in the other free space btree?
+
+4. Inode btree records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Is there a matching record in free inode btree?
+
+   - Do cleared bits in the holemask correspond with inode

clusters?
+
+   - Do set bits in the freemask correspond with inode records with
zero link
+     count?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+5. Inode records are cross-referenced as follows:
+
+   - Everything in class 1.
+
+   - Do all the fields that summarize information about the file

forks actually
+     match those forks?
+
+   - Does each inode with zero link count correspond to a record in
the free
+     inode btree?
+
+6. File fork space mapping records are cross-referenced as follows:

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+   - Everything in class 1 and 2 above.
+
+   - Is this space not mentioned by the inode btrees?
+
+   - If this is a CoW fork mapping, does it correspond to a CoW

entry in the
+     reference count btree?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+7. Reference count records are cross-referenced as follows:
+
+   - Everything in class 1 and 2 above.
+
+   - Within the space subkeyspace of the rmap btree (that is to

say,
all
+     records mapped to a particular space extent and ignoring the
owner info),
+     are there the same number of reverse mapping records for each
block as the
+     reference count record claims?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+Proposed patchsets are the series to find gaps in
+`refcount btree
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-refcount-gaps>`_,
+`inode btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-inobt-gaps>`_, and
+`rmap btree
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-rmapbt-gaps>`_ records;
+to find
+`mergeable records
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-detect-mergeable-records>`_;
+and to
+`improve cross referencing with rmap
+<
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-strengthen-rmap-checking>`_

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+before starting a repair.
+
+Checking Extended Attributes
+````````````````````````````
+
+Extended attributes implement a key-value store that enable

fragments of data

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+to be attached to any file.
+Both the kernel and userspace can access the keys and values,

subject to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+namespace and privilege restrictions.
+Most typically these fragments are metadata about the file --

origins, security

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+contexts, user-supplied labels, indexing information, etc.
+
+Names can be as long as 255 bytes and can exist in several

different

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+namespaces.
+Values can be as large as 64KB.
+A file's extended attributes are stored in blocks mapped by the

attr
fork.
+The mappings point to leaf blocks, remote value blocks, or dabtree
blocks.
+Block 0 in the attribute fork is always the top of the structure,
but otherwise
+each of the three types of blocks can be found at any offset in the
attr fork.
+Leaf blocks contain attribute key records that point to the name and
the value.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+Names are always stored elsewhere in the same leaf block.
+Values that are less than 3/4 the size of a filesystem block are

also stored

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+elsewhere in the same leaf block.
+Remote value blocks contain values that are too large to fit

inside
a leaf.
+If the leaf information exceeds a single filesystem block, a dabtree
(also
+rooted at block 0) is created to map hashes of the attribute names
to leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the attr fork.
+
+Checking an extended attribute structure is not so

straightfoward
due to the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+lack of separation between attr blocks and index blocks.
+Scrub must read each block mapped by the attr fork and ignore

the
non-leaf

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks:
+
+1. Walk the dabtree in the attr fork (if present) to ensure that

there are no
+   irregularities in the blocks or dabtree mappings that do not point to
+   attr leaf blocks.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+2. Walk the blocks of the attr fork looking for leaf blocks.
+   For each entry inside a leaf:
+
+   a. Validate that the name does not contain invalid

characters.
+
+   b. Read the attr value.
+      This performs a named lookup of the attr name to ensure the
correctness
+      of the dabtree.
+      If the value is stored in a remote block, this also validates
the
+      integrity of the remote value block.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Checking and Cross-Referencing Directories
+``````````````````````````````````````````
+
+The filesystem directory tree is a directed acylic graph

structure,
with files
+constituting the nodes, and directory entries (dirents) constituting
the edges.
+Directories are a special type of file containing a set of mappings
from a
+255-byte sequence (name) to an inumber.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+These are called directory entries, or dirents for short.
+Each directory file must have exactly one directory pointing to

the
file.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+A root directory points to itself.
+Directory entries point to files of any type.
+Each non-directory file may have multiple directories point to

it.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+In XFS, directories are implemented as a file containing up to

three
32GB

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+partitions.
+The first partition contains directory entry data blocks.
+Each data block contains variable-sized records associating a

user-
provided

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+name with an inumber and, optionally, a file type.
+If the directory entry data grows beyond one block, the second

partition (which
+exists as post-EOF extents) is populated with a block containing free space
+information and an index that maps hashes of the dirent names to directory data

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+blocks in the first partition.
+This makes directory name lookups very fast.
+If this second partition grows beyond one block, the third

partition
is
+populated with a linear array of free space information for faster

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+expansions.
+If the free space has been separated and the second partition

grows
again
+beyond one block, then a dabtree is used to map hashes of dirent names to

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+directory data blocks.
+
+Checking a directory is pretty straightfoward:
+
+1. Walk the dabtree in the second partition (if present) to

ensure
that there
+   are no irregularities in the blocks or dabtree mappings that do
not point to
+   dirent blocks.
+
+2. Walk the blocks of the first partition looking for directory entries.
+   Each dirent is checked as follows:

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+
+   a. Does the name contain no invalid characters?
+
+   b. Does the inumber correspond to an actual, allocated inode?
+
+   c. Does the child inode have a nonzero link count?
+
+   d. If a file type is included in the dirent, does it match

the
type of the
+      inode?
+
+   e. If the child is a subdirectory, does the child's dotdot pointer point
+      back to the parent?
+
+   f. If the directory has a second partition, perform a named lookup of the
+      dirent name to ensure the correctness of the dabtree. +
+3. Walk the free space list in the third partition (if present) to
ensure that
+   the free spaces it describes are really unused.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+Checking operations involving :ref:`parents <dirparent>` and
+:ref:`file link counts <nlinks>` are discussed in more detail in

later

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+sections.
+
+Checking Directory/Attribute Btrees
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As stated in previous sections, the directory/attribute btree

(dabtree) index
+maps user-provided names to improve lookup times by avoiding linear
scans.
+Internally, it maps a 32-bit hash of the name to a block offset within the

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+appropriate file fork.
+
+The internal structure of a dabtree closely resembles the btrees

that record
+fixed-size metadata records -- each dabtree block contains a magic
number, a
+checksum, sibling pointers, a UUID, a tree level, and a log sequence
number.
+The format of leaf and node records are the same -- each entry points to the
+next level down in the hierarchy, with dabtree node records pointing
to dabtree
+leaf blocks, and dabtree leaf records pointing to non-dabtree blocks
elsewhere

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+in the fork.
+
+Checking and cross-referencing the dabtree is very similar to

what
is done for

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+space btrees:
+
+- Does the type of data stored in the block match what scrub is

expecting?
+
+- Does the block belong to the owning structure that asked for the
read?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Do the records fit within the block?
+
+- Are the records contained inside the block free of obvious

corruptions?

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+- Are the name hashes in the correct order?
+
+- Do node pointers within the dabtree point to valid fork

offsets
for dabtree
+  blocks?
+
+- Do leaf pointers within the dabtree point to valid fork offsets
for directory
+  or attr leaf blocks?

Documentation/filesystems/xfs-online-fsck-design.rst +6 -0
+
+- Do child pointers point towards the leaves?
+
+- Do sibling pointers point across the same level?
+
+- For each dabtree node record, does the record key accurate

reflect
the
+  contents of the child dabtree block?
+
+- For each dabtree leaf record, does the record key accurate reflect
the
+  contents of the directory or attr block?

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Cross-Referencing Summary Counters
+``````````````````````````````````
+
+XFS maintains three classes of summary counters: available

resources, quota

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+resource usage, and file link counts.
+
+In theory, the amount of available resources (data blocks,

inodes,
realtime

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+extents) can be found by walking the entire filesystem.
+This would make for very slow reporting, so a transactional

filesystem can

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+maintain summaries of this information in the superblock.
+Cross-referencing these values against the filesystem metadata

should be a
+simple matter of walking the free space and inode metadata in each
AG and the
+realtime bitmap, but there are complications that will be discussed
in
+:ref:more detail <fscounters> later.
+
+:ref:Quota usage <quotacheck> and :ref:file link count <nlinks>
+checking are sufficiently complicated to warrant separate sections.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Post-Repair Reverification
+``````````````````````````
+
+After performing a repair, the checking code is run a second

time to
validate
+the new structure, and the results of the health assessment are recorded

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+internally and returned to the calling process.
+This step is critical for enabling system administrator to

monitor
the status

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the filesystem and the progress of any repairs.
+For developers, it is a useful means to judge the efficacy of

error
detection

Documentation/filesystems/xfs-online-fsck-design.rst +1 -0
+and correction in the online and offline checking tools.
(quoted diff) +0 -0

metadata.rst b/Documentation/filesystems/xfs-self-describing- metadata.rst
index b79dbf36dc94..a10c4ae6955e 100644

Documentation/filesystems/xfs-self-describing-metadata.rst +0 -0
@@ -1,4 +1,5 @@

 .. SPDX-License-Identifier: GPL-2.0
+.. _xfs_self_describing_metadata:

 ============================
 XFS Self Describing Metadata

#24 lore raw

PATCH 6/14 xfs: document how online fsck deals with eventual consistency

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model to break up complex multistep metadata updates into small chained transactions. This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action unless it can be shown that there are no other threads processing a transaction chain. This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303 ++++++++++++++++++++
 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +303 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata
+inconsistency.
+The root cause of these reports is the eventual consistency model introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.
+
+Originally, transaction chains were added to XFS to avoid deadlocks when
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
+items to commit to freeing some space in one transaction while deferring the
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's block mapping
+   structures to remove the mapping from the btree blocks.
+   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct
+   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space.
+   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.
+
+If the system goes down after transaction #1 is written back to the filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when recovery finds
+an intent log item but does not find a corresponding intent done item, it will
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in the recovered
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to consider.
+The first is that log items must be added to a transaction in the correct order
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block must be
+completed before the last update to free the extent, and extents should not
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are (usually)
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate state,
+but as long as the first subtlety is handled, this should not affect the
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which means that
+offline fsck never sees the temporary inconsistencies caused by deferred work
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid deadlocks
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features, it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other file
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per transaction
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be done once to
+remove the space from a staging area and again to map it into the file!
+
+To deal with this explosion in a calm manner, XFS expands its use of deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.
+
+However, online fsck changes the rules -- remember that although physical
+updates to per-AG structures are coordinated by locking the buffers for AG
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure, it must do
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock, scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be catastrophic!
+
+Several solutions to this problem were evaluated upon discovery of this flaw:
+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items.
+   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.
+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.
+
+Intent Drains
+`````````````
+
+The fourth solution is implemented in the current iteration of online fsck,
+with atomic_t providing the active intent counter.
+
+There are two key properties to the drain mechanism.
+First, the counter is incremented when a deferred work item is *queued* to a
+transaction, and it is decremented after the associated intent done log item is
+*committed* to another transaction.
+The second property is that deferred work can be added to a transaction without
+holding an AG header lock, but per-AG work items cannot be marked done without
+locking that AG header buffer to log the physical updates and the intent done
+log item.
+The first property enables scrub to yield to running transaction chains, which
+is an explicit deprioritization of online fsck to benefit file operations.
+The second property of the drain is key to the correct coordination of scrub,
+since scrub will always be able to decide if a conflict is possible.
+
+For regular filesystem code, the drain works as follows:
+
+1. Call the appropriate subsystem function to add a deferred work item to a
+   transaction.
+
+2. The function calls ``xfs_drain_bump`` to increase the counter.
+
+3. When the deferred item manager wants to finish the deferred work item, it
+   calls ``->finish_item`` to complete it.
+
+4. The ``->finish_item`` implementation logs some changes and calls
+   ``xfs_drain_drop`` to decrease the sloppy counter and wake up any threads
+   waiting on the drain.
+
+5. The subtransaction commits, which unlocks the resource associated with the
+   intent item.
+
+For scrub, the drain works as follows:
+
+1. Lock the resource(s) associated with the metadata being scrubbed.
+   For example, a scan of the refcount btree would lock the AGI and AGF header
+   buffers.
+
+2. If the counter is zero (``xfs_drain_busy`` returns false), there are no
+   chains in progress and the operation may proceed.
+
+3. Otherwise, release the resources grabbed in step 1.
+
+4. Wait for the intent counter to reach zero (``xfs_drain_intents``), then go
+   back to step 1 unless a signal has been caught.
+
+To avoid polling in step 4, the drain provides a waitqueue for scrub threads to
+be woken up whenever the intent count drops to zero.
+
+The proposed patchset is the
+`scrub intent drain series
+<https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-drain-intents>`_.
+
+.. _jump_labels:
+
+Static Keys (aka Jump Label Patching)
+`````````````````````````````````````
+
+Online fsck for XFS separates the regular filesystem from the checking and
+repair code as much as possible.
+However, there are a few parts of online fsck (such as the intent drains, and
+later, live update hooks) where it is useful for the online fsck code to know
+what's going on in the rest of the filesystem.
+Since it is not expected that online fsck will be constantly running in the
+background, it is very important to minimize the runtime overhead imposed by
+these hooks when online fsck is compiled into the kernel but not actively
+running on behalf of userspace.
+Taking locks in the hot path of a writer thread to access a data structure only
+to find that no further action is necessary is expensive -- on the author's
+computer, this have an overhead of 40-50ns per access.
+Fortunately, the kernel supports dynamic code patching, which enables XFS to
+replace a static branch to hook code with ``nop`` sleds when online fsck isn't
+running.
+This sled has an overhead of however long it takes the instruction decoder to
+skip past the sled, which seems to be on the order of less than 1ns and
+does not access memory outside of instruction fetching.
+
+When online fsck enables the static key, the sled is replaced with an
+unconditional branch to call the hook code.
+The switchover is quite expensive (~22000ns) but is paid entirely by the
+program that invoked online fsck, and can be amortized if multiple threads
+enter online fsck at the same time, or if multiple filesystems are being
+checked at the same time.
+Changing the branch direction requires taking the CPU hotplug lock, and since
+CPU initialization requires memory allocation, online fsck must be careful not
+to change a static key while holding any locks or resources that could be
+accessed in the memory reclaim paths.
+To minimize contention on the CPU hotplug lock, care should be taken not to
+enable or disable static keys unnecessarily.
+
+Because static keys are intended to minimize hook overhead for regular
+filesystem operations when xfs_scrub is not running, the intended usage
+patterns are as follows:
+
+- The hooked part of XFS should declare a static-scoped static key that
+  defaults to false.
+  The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this.
+  The static key itself should be declared as a ``static`` variable.
+
+- When deciding to invoke code that's only used by scrub, the regular
+  filesystem should call the ``static_branch_unlikely`` predicate to avoid the
+  scrub-only hook code if the static key is not enabled.
+
+- The regular filesystem should export helper functions that call
+  ``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
+  static key.
+  Wrapper functions make it easy to compile out the relevant code if the kernel
+  distributor turns off online fsck at build time.
+
+- Scrub functions wanting to turn on scrub-only XFS functionality should call
+  the ``xchk_fshooks_enable`` from the setup function to enable a specific
+  hook.
+  This must be done before obtaining any resources that are used by memory
+  reclaim.
+  Callers had better be sure they really need the functionality gated by the
+  static key; the ``TRY_HARDER`` flag is useful here.
+
+Online scrub has resource acquisition helpers (e.g. ``xchk_perag_lock``) to
+handle locking AGI and AGF buffers for all scrubber functions.
+If it detects a conflict between scrub and the running transactions, it will
+try to wait for intents to complete.
+If the caller of the helper has not enabled the static key, the helper will
+return -EDEADLOCK, which should result in the scrub being restarted with the
+``TRY_HARDER`` flag set.
+The scrub setup function should detect that flag, enable the static key, and
+try the scrub again.
+Scrub teardown disables all static keys obtained by ``xchk_fshooks_enable``.
+
+For more information, please see the kernel documentation of
+Documentation/staging/static-keys.rst.
#25 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 185 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model to break up complex multistep metadata updates into small chained transactions. This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action unless it can be shown that there are no other threads processing a transaction chain. This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303 ++++++++++++++++++++
 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +159 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata
+inconsistency.
+The root cause of these reports is the eventual consistency model introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.
+
+Originally, transaction chains were added to XFS to avoid deadlocks when
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
+items to commit to freeing some space in one transaction while deferring the
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's block mapping
+   structures to remove the mapping from the btree blocks.
+   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct
+   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space.
+   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.
+
+If the system goes down after transaction #1 is written back to the filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when recovery finds
+an intent log item but does not find a corresponding intent done item, it will
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in the recovered
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to consider.
+The first is that log items must be added to a transaction in the correct order
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block must be
+completed before the last update to free the extent, and extents should not
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are (usually)
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate state,
+but as long as the first subtlety is handled, this should not affect the
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which means that
+offline fsck never sees the temporary inconsistencies caused by deferred work
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid deadlocks
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features, it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other file
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per transaction
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be done once to
+remove the space from a staging area and again to map it into the file!
+
+To deal with this explosion in a calm manner, XFS expands its use of deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.
+
+However, online fsck changes the rules -- remember that although physical
+updates to per-AG structures are coordinated by locking the buffers for AG
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure, it must do
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock, scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be catastrophic!
+
+Several solutions to this problem were evaluated upon discovery of this flaw:
+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items.
+   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.
+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status · line 1614

Quoted 10 lines
Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.
+

Is there any guarantee that some silly real life regular filesystem workload won't starve online fsck forever?
IOW, is forward progress of online fsck guaranteed?

Good luck with landing online fsck before the 2024 NYE deluge ;)

Thanks,
Amir.

#26 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 192 lines · Amir Goldstein
Quoted 191 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model to break up complex multistep metadata updates into small chained transactions. This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action unless it can be shown that there are no other threads processing a transaction chain. This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303 ++++++++++++++++++++
 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +165 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata
+inconsistency.
+The root cause of these reports is the eventual consistency model introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.
+
+Originally, transaction chains were added to XFS to avoid deadlocks when
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
+items to commit to freeing some space in one transaction while deferring the
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's block mapping
+   structures to remove the mapping from the btree blocks.
+   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct
+   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space.
+   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.
+
+If the system goes down after transaction #1 is written back to the filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when recovery finds
+an intent log item but does not find a corresponding intent done item, it will
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in the recovered
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to consider.
+The first is that log items must be added to a transaction in the correct order
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block must be
+completed before the last update to free the extent, and extents should not
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are (usually)
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate state,
+but as long as the first subtlety is handled, this should not affect the
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which means that
+offline fsck never sees the temporary inconsistencies caused by deferred work
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid deadlocks
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features, it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other file
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per transaction
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be done once to
+remove the space from a staging area and again to map it into the file!
+
+To deal with this explosion in a calm manner, XFS expands its use of deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.
+
+However, online fsck changes the rules -- remember that although physical
+updates to per-AG structures are coordinated by locking the buffers for AG
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure, it must do
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock, scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be catastrophic!
+
+Several solutions to this problem were evaluated upon discovery of this flaw:
+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items.
+   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.
+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status · line 1614

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.
+

Is there any guarantee that some silly real life regular filesystem workload won't starve online fsck forever?
IOW, is forward progress of online fsck guaranteed?

Nope, forward progress isn't guaranteed. The kernel checks for fatal signals every time it backs off a scrub so at least we don't end up with unkillable processes. At one point I added a timeout field to the ioctl interface so that the kernel could time out an operation if it took too long to acquire the necessary resources. So far, the "race fsstress and xfs_scrub" tests have not shown scrub failing to make any forward progress.

That said, I have /not/ yet had a chance to try it out any of these massive 1000-core systems with an according workload.

Quoted 1 line

Good luck with landing online fsck before the 2024 NYE deluge ;)

Thank *you* for reading this chapter of the design document!! :)

--D

Quoted 2 lines

Thanks,
Amir.

#27 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 203 lines · Darrick J. Wong
Quoted 192 lines · Amir Goldstein
Quoted 191 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model to break up complex multistep metadata updates into small chained transactions. This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action unless it can be shown that there are no other threads processing a transaction chain. This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303 ++++++++++++++++++++
 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +165 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata
+inconsistency.
+The root cause of these reports is the eventual consistency model introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.
+
+Originally, transaction chains were added to XFS to avoid deadlocks when
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
+items to commit to freeing some space in one transaction while deferring the
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's block mapping
+   structures to remove the mapping from the btree blocks.
+   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct
+   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space.
+   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.
+
+If the system goes down after transaction #1 is written back to the filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when recovery finds
+an intent log item but does not find a corresponding intent done item, it will
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in the recovered
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to consider.
+The first is that log items must be added to a transaction in the correct order
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block must be
+completed before the last update to free the extent, and extents should not
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are (usually)
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate state,
+but as long as the first subtlety is handled, this should not affect the
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which means that
+offline fsck never sees the temporary inconsistencies caused by deferred work
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid deadlocks
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features, it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other file
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per transaction
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be done once to
+remove the space from a staging area and again to map it into the file!
+
+To deal with this explosion in a calm manner, XFS expands its use of deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.
+
+However, online fsck changes the rules -- remember that although physical
+updates to per-AG structures are coordinated by locking the buffers for AG
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure, it must do
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock, scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be catastrophic!
+
+Several solutions to this problem were evaluated upon discovery of this flaw:
+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items.
+   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.
+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
Quoted 8 lines

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status · line 1614

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.
+

Is there any guarantee that some silly real life regular filesystem workload won't starve online fsck forever?
IOW, is forward progress of online fsck guaranteed?

Quoted 1 line

Nope, forward progress isn't guaranteed.

That sounds like a problem.

Quoted 11 lines

The kernel checks for fatal
signals every time it backs off a scrub so at least we don't end up with unkillable processes. At one point I added a timeout field to the ioctl interface so that the kernel could time out an operation if it took too long to acquire the necessary resources. So far, the "race fsstress and xfs_scrub" tests have not shown scrub failing to make any forward progress.

That said, I have /not/ yet had a chance to try it out any of these massive 1000-core systems with an according workload.

Don't know if fsstress is the best way to check the worst case scenario.

Can you think of a workload, say several threads creating and deleting temp files, with deferred parent pointer items preventing the queue from ever draining?

Considering that a "full journal" scenario is always going to be a possible worst case incident, how bad would it be to block new transactions instead of the possibility of starving scrub consistency checks forever?

Wouldn't the consistency checks be much faster than freeing journal space would be in a "full journal" situation?

I don't know if there is a "mission statement" for online fsck, but I think it would say "minimal user interference" not "no user interference". It sounds like the interference we are trying to avoid is light years away from the downtime of offline fsck, so online fsck would still be a huge win. online fsck that never ends OTOH... maybe less so.

Quoted 4 lines
Quoted 1 line

Good luck with landing online fsck before the 2024 NYE deluge ;)

Thank *you* for reading this chapter of the design document!! :)

Oh I read them all at the summer submission, but it took me so long that I forgot to follow up..

My other question was regarding memory usage control. I have horrid memories from e2fsck unpredictable memory usage and unpredictable runtime due to swapping.

xfs_repair -m was a huge improvement compared to e2fsck.
I don't remember reading about memory usage limits for online repair, so I was concerned about unpredictable memory usage and swapping. Can you say something to ease those concerns?

Thanks,
Amir.

#28 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 204 lines · Amir Goldstein
Quoted 203 lines · Darrick J. Wong
Quoted 200 lines · Amir Goldstein
Quoted 185 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model to break up complex multistep metadata updates into small chained transactions. This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action unless it can be shown that there are no other threads processing a transaction chain. This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303 ++++++++++++++++++++
 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +159 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata
+inconsistency.
+The root cause of these reports is the eventual consistency model introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.
+
+Originally, transaction chains were added to XFS to avoid deadlocks when
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent (EFI) log
+items to commit to freeing some space in one transaction while deferring the
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's block mapping
+   structures to remove the mapping from the btree blocks.
+   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct
+   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space.
+   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.
+
+If the system goes down after transaction #1 is written back to the filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when recovery finds
+an intent log item but does not find a corresponding intent done item, it will
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in the recovered
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to consider.
+The first is that log items must be added to a transaction in the correct order
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block must be
+completed before the last update to free the extent, and extents should not
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are (usually)
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate state,
+but as long as the first subtlety is handled, this should not affect the
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which means that
+offline fsck never sees the temporary inconsistencies caused by deferred work
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid deadlocks
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features, it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other file
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per transaction
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be done once to
+remove the space from a staging area and again to map it into the file!
+
+To deal with this explosion in a calm manner, XFS expands its use of deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.
+
+However, online fsck changes the rules -- remember that although physical
+updates to per-AG structures are coordinated by locking the buffers for AG
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure, it must do
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock, scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be catastrophic!
+
+Several solutions to this problem were evaluated upon discovery of this flaw:
+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items.
+   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.
+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status · line 1614

Quoted 10 lines
Documentation/filesystems/xfs-online-fsck-design.rst +10 -0
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.
+

Is there any guarantee that some silly real life regular filesystem workload won't starve online fsck forever?
IOW, is forward progress of online fsck guaranteed?

Quoted 3 lines
Quoted 1 line

Nope, forward progress isn't guaranteed.

That sounds like a problem.

So far it hasn't been. I prefer to sacrifice performance of the background fsck service for the sake of foreground tasks. The fsstress and fsx fstests haven't shown any particularly serious issues. I've also kicked off xfs_scrub on the same VM hosts that are running the fuzz test suite (~52 VMs per host) and scrub can still finish the filesystem in a couple of hours.

Things get markedly worse on spinning rust with a lot of parallel unwritten extent conversions and allocations going on (aka the disk backup systems). Normally a backup from flash to rust takes about an hour; with scrub and backup contending for the head actuator, it'll go up to about 2-3 hours, but both tasks can make (verrrry slow) forward progress.

That said -- the backup program spends a lot of iowait time waiting for file data blocks to read in or get written back, so the contention is on the storage hardware, not the filesystem locks.

Quoted 12 lines
Quoted 11 lines

The kernel checks for fatal
signals every time it backs off a scrub so at least we don't end up with unkillable processes. At one point I added a timeout field to the ioctl interface so that the kernel could time out an operation if it took too long to acquire the necessary resources. So far, the "race fsstress and xfs_scrub" tests have not shown scrub failing to make any forward progress.

That said, I have /not/ yet had a chance to try it out any of these massive 1000-core systems with an according workload.

Quoted 5 lines

Don't know if fsstress is the best way to check the worst case scenario.

Can you think of a workload, say several threads creating and deleting temp files, with deferred parent pointer items preventing the queue from ever draining?

The worst workload would be one that is entirely metadata based -- a giant directory tree full of empty files with all information being stored as extended attributes.

Quoted 3 lines

Considering that a "full journal" scenario is always going to be a possible worst case incident, how bad would it be to block new transactions instead of the possibility of starving scrub consistency checks forever?

First of all, scrub has already allocated a transaction by the time it gets to the intent drain step. There's no good way to block new transactions once we've reached this stage, nor should there be. Blocking transactions stalls xfs garbage collection and memory reclaim.

Quoted 2 lines

Wouldn't the consistency checks be much faster than freeing journal space would be in a "full journal" situation?

I haven't investigated this in depth, but yes, scrub should be faster than forcing the log and checkpointing the log to move the log tail forward to empty out the journal.

Quoted 2 lines

I don't know if there is a "mission statement" for online fsck, but I think it would say "minimal user interference" not "no user interference".

Yes. The section about eventual consistency states that "...regular filesystem updates take precedence over online fsck activity".

Quoted 3 lines

It sounds like the interference we are trying to avoid is light years away from the downtime of offline fsck, so online fsck would still be a huge win. online fsck that never ends OTOH... maybe less so.

Well you /can/ just kill the xfs_scrub processes if they are taking too much time. One of the nastier papercuts of the background scrub is that the fs cannot be unmounted while it's running, and systemd doesn't have a good mechanism for "kill this service before stopping this mount". Or maybe it does and I haven't yet found it?

(The cronjob variant definitely suffers from that...)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Good luck with landing online fsck before the 2024 NYE deluge ;)

Thank *you* for reading this chapter of the design document!! :)

Oh I read them all at the summer submission, but it took me so long that I forgot to follow up..

Yeah, that seems to be a common problem with large new features. :/

Quoted 8 lines

My other question was regarding memory usage control. I have horrid memories from e2fsck unpredictable memory usage and unpredictable runtime due to swapping.

xfs_repair -m was a huge improvement compared to e2fsck.
I don't remember reading about memory usage limits for online repair, so I was concerned about unpredictable memory usage and swapping. Can you say something to ease those concerns?

Both e2fsck and xfs_repair have to be capable of repairing the entire filesystem all at once, which means that they allocate many many of incore objects from which all of the ondisk space metadata (ag btrees in the case of xfs, bitmaps for e2fsck) is regenerated. Since the fs is offline, it's considered advantageous to perform *one* scan and rebuild everything all at once, even if the memory cost is high.

xfs_scrub scans and repairs each metadata object individually, which means that it only needs to allocate as much (kernel/xfile) memory as needed to scan a single btree/inode record/quota record/bitmap. For scans the memory requirements are usually minimal since it creates a bunch of btree cursors and cross-references records.

For repairs, the memory requirements are on the order of the size of the new data structure that will be written out. We scan the fs to build the new recordset in memory, compute the size of the new btree, allocate some blocks, and format the records into the blocks before committing the btree root.

For summary data (e.g. link counts, dquots) we build a shadow copy in memory, so the memory requirements are on the order of the number of files in the fs and the number of uid/gid/projid in the filesystem, respectively.

Most of the intermediate structures are stuffed into a tmpfs file, which means they can be paged out to disk. If there's really no memory available, scrub can abort all the way out to userspace provided it hasn't committed anything to disk yet.

IOWs, online fsck generally only requires enough memory to build a new copy of whichever objects it happens to be scanning at any given moment. The background service runs single-threaded to avoid consuming a lot of CPU or memory.

--D

Quoted 2 lines

Thanks,
Amir.

#29 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

In reply to Darrick J. Wong
Quoted 227 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model
to break up complex multistep metadata updates into small chained transactions.  This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action
unless it can be shown that there are no other threads processing a transaction chain.  This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303
++++++++++++++++++++

 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system

administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress

tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistency.
+The root cause of these reports is the eventual consistency model

introduced by
+the expansion of deferred work items and compound transaction chains when

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+reverse mapping and reflink were introduced.
 
 
 
+
+Originally, transaction chains were added to XFS to avoid deadlocks

when

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in

increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent

(EFI) log
+items to commit to freeing some space in one transaction while deferring the

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's

block mapping
+   structures to remove the mapping from the btree blocks. +   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct +   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order. +
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space. +   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+If the system goes down after transaction #1 is written back to the

filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when

recovery finds
+an intent log item but does not find a corresponding intent done item, it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in

the recovered

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to

consider.
+The first is that log items must be added to a transaction in the correct order

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block

must be
+completed before the last update to free the extent, and extents should not

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are

(usually)

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate

state,
+but as long as the first subtlety is handled, this should not affect the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which

means that
+offline fsck never sees the temporary inconsistencies caused by deferred work

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid

deadlocks

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features,

it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other

file

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per

transaction

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be

done once to
+remove the space from a staging area and again to map it into the file!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To deal with this explosion in a calm manner, XFS expands its use of

deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its

deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+However, online fsck changes the rules -- remember that although

physical
+updates to per-AG structures are coordinated by locking the buffers for AG

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure,

it must do

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock,

scrub may have

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a

reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be

catastrophic!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Several solutions to this problem were evaluated upon discovery of
Quoted 1 line

this flaw:

Hmm, so while having a really in depth efi example is insightful, I wonder if it would be more oranized to put it in a separate document somewhere and just reference it. As far as ofsck is concerned, I think a lighter sumary would do:

"Complex operations that modify multiple AGs are performed through a series of transactions which are logged to a journal that an offline fsck can either replay or discard. Online fsck however, must be able to deal with these operations while they are still in progress. This presents a unique challenge for ofsck since a partially completed transaction chain may present the appearance of inconsistencies, even though the operations are functioning as intended. (For a more detailed example, see <cite document here...>)

The challenge then becomes how to avoid incorrectly repairing these non-issues as doing so would cause more harm than help."

Quoted 19 lines

+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.

Quoted 8 lines

+   This would introduce a lot of complexity into the coordinator since it is
+   only loosely coupled with the actual deferred work items. +   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.

Hmm, that one doesnt seem like it's really an option then :-(

Quoted 21 lines

+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG. +   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify

Quoted 8 lines

that AG, and
+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.

So basically it sounds like 4 is the only reasonable option? If the discussion concerning the other options have died down, I would clean them out. They're great for brain storming and invitations for collaboration, but ideally the goal of any of that should be to narrow down an agreed upon plan of action. And the goal of your document should make clear what that plan is. So if no one has any objections by now, maybe just tie it right into the last line:

"The challenge then becomes how to avoid incorrectly repairing these non-issues as doing so would cause more harm than help. Fortunately only online fsck has this requirement of total consistency..."

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Intent Drains
+`````````````
+
+The fourth solution is implemented in the current iteration of

This solution is implemented...

Quoted 63 lines

online fsck,

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+with atomic_t providing the active intent counter.
+
+There are two key properties to the drain mechanism.
+First, the counter is incremented when a deferred work item is

*queued* to a
+transaction, and it is decremented after the associated intent done log item is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+*committed* to another transaction.
+The second property is that deferred work can be added to a

transaction without
+holding an AG header lock, but per-AG work items cannot be marked done without
+locking that AG header buffer to log the physical updates and the intent done

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+log item.
+The first property enables scrub to yield to running transaction

chains, which
+is an explicit deprioritization of online fsck to benefit file operations.
+The second property of the drain is key to the correct coordination of scrub,

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+since scrub will always be able to decide if a conflict is possible.
+
+For regular filesystem code, the drain works as follows:
+
+1. Call the appropriate subsystem function to add a deferred work

item to a
+   transaction.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+2. The function calls ``xfs_drain_bump`` to increase the counter.
+
+3. When the deferred item manager wants to finish the deferred work

item, it
+   calls ``->finish_item`` to complete it.
+
+4. The ``->finish_item`` implementation logs some changes and calls +   ``xfs_drain_drop`` to decrease the sloppy counter and wake up any threads
+   waiting on the drain.
+
+5. The subtransaction commits, which unlocks the resource associated with the
+   intent item.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+For scrub, the drain works as follows:
+
+1. Lock the resource(s) associated with the metadata being scrubbed.
+   For example, a scan of the refcount btree would lock the AGI and

AGF header
+   buffers.
+
+2. If the counter is zero (``xfs_drain_busy`` returns false), there are no
+   chains in progress and the operation may proceed.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+3. Otherwise, release the resources grabbed in step 1.
+
+4. Wait for the intent counter to reach zero

(``xfs_drain_intents``), then go
+   back to step 1 unless a signal has been caught.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To avoid polling in step 4, the drain provides a waitqueue for scrub
Quoted 2 lines

threads to
+be woken up whenever the intent count drops to zero.

I think all that makes sense

Quoted 117 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+The proposed patchset is the
+`scrub intent drain series
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-drain-intents>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +7 -0
+
+.. _jump_labels:
+
+Static Keys (aka Jump Label Patching)
+`````````````````````````````````````
+
+Online fsck for XFS separates the regular filesystem from the

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repair code as much as possible.
+However, there are a few parts of online fsck (such as the intent

drains, and
+later, live update hooks) where it is useful for the online fsck code to know

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+what's going on in the rest of the filesystem.
+Since it is not expected that online fsck will be constantly running

in the
+background, it is very important to minimize the runtime overhead imposed by
+these hooks when online fsck is compiled into the kernel but not actively

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running on behalf of userspace.
+Taking locks in the hot path of a writer thread to access a data

structure only
+to find that no further action is necessary is expensive -- on the author's

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer, this have an overhead of 40-50ns per access.
+Fortunately, the kernel supports dynamic code patching, which

enables XFS to
+replace a static branch to hook code with ``nop`` sleds when online fsck isn't

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running.
+This sled has an overhead of however long it takes the instruction

decoder to
+skip past the sled, which seems to be on the order of less than 1ns and

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+does not access memory outside of instruction fetching.
+
+When online fsck enables the static key, the sled is replaced with

an

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unconditional branch to call the hook code.
+The switchover is quite expensive (~22000ns) but is paid entirely by

the
+program that invoked online fsck, and can be amortized if multiple threads
+enter online fsck at the same time, or if multiple filesystems are being

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+checked at the same time.
+Changing the branch direction requires taking the CPU hotplug lock,

and since
+CPU initialization requires memory allocation, online fsck must be careful not
+to change a static key while holding any locks or resources that could be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+accessed in the memory reclaim paths.
+To minimize contention on the CPU hotplug lock, care should be taken

not to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+enable or disable static keys unnecessarily.
+
+Because static keys are intended to minimize hook overhead for

regular
+filesystem operations when xfs_scrub is not running, the intended usage

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+patterns are as follows:
+
+- The hooked part of XFS should declare a static-scoped static key

that
+  defaults to false.
+  The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this. +  The static key itself should be declared as a ``static`` variable. +
+- When deciding to invoke code that's only used by scrub, the regular
+  filesystem should call the ``static_branch_unlikely`` predicate to avoid the
+  scrub-only hook code if the static key is not enabled. +
+- The regular filesystem should export helper functions that call +  ``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
+  static key.
+  Wrapper functions make it easy to compile out the relevant code if the kernel
+  distributor turns off online fsck at build time.
+
+- Scrub functions wanting to turn on scrub-only XFS functionality should call
+  the ``xchk_fshooks_enable`` from the setup function to enable a specific
+  hook.
+  This must be done before obtaining any resources that are used by memory
+  reclaim.
+  Callers had better be sure they really need the functionality gated by the
+  static key; the ``TRY_HARDER`` flag is useful here.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Online scrub has resource acquisition helpers (e.g.

``xchk_perag_lock``) to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+handle locking AGI and AGF buffers for all scrubber functions.
+If it detects a conflict between scrub and the running transactions,

it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try to wait for intents to complete.
+If the caller of the helper has not enabled the static key, the

helper will
+return -EDEADLOCK, which should result in the scrub being restarted with the
+``TRY_HARDER`` flag set.
+The scrub setup function should detect that flag, enable the static key, and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try the scrub again.
+Scrub teardown disables all static keys obtained by
Quoted 1 line

``xchk_fshooks_enable``.

Ok, this part here seems pretty well documented. Organizing nits aside I think it looks good.

Allison

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+For more information, please see the kernel documentation of
+Documentation/staging/static-keys.rst.
#30 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 40 lines · Allison Henderson
Quoted 39 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model
to break up complex multistep metadata updates into small chained transactions.  This is generally good for performance and scalability because XFS doesn't need to prepare for enormous transactions, but it also means that online fsck must be careful not to attempt a fsck action
unless it can be shown that there are no other threads processing a transaction chain.  This part of the design documentation covers the thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303
++++++++++++++++++++

 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system

administrator to monitor the status
 of the filesystem and the progress of any repairs.
 For developers, it is a useful means to judge the efficacy of error detection
 and correction in the online and offline checking tools.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress

tests
+uncovered a misinteraction between online fsck and compound transaction chains

Quoted 11 lines
Quoted 8 lines

+created by other writer threads that resulted in false reports of metadata

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistency.
+The root cause of these reports is the eventual consistency model

introduced by
+the expansion of deferred work items and compound transaction chains when
+reverse mapping and reflink were introduced.

Was there supposed to be a comment here?

Quoted 195 lines
Quoted 177 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Originally, transaction chains were added to XFS to avoid deadlocks

when

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in

increasing order,
+which makes it impossible (say) to use a single transaction to free a space
+extent in AG 7 and then try to free a now superfluous block mapping btree block

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing Intent

(EFI) log
+items to commit to freeing some space in one transaction while deferring the

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the file's

block mapping
+   structures to remove the mapping from the btree blocks. +   It then attaches to the in-memory transaction an action item to schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct +   xfs_defer_pending`` objects, each of which maintains a list of ``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT) block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order. +
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical update to the
+   free space btrees of AG 7 to release the unmapped file space. +   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so that log
+   recovery can tell if the EFI needs to be replayed.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+If the system goes down after transaction #1 is written back to the

filesystem
+but before #2 is committed, a scan of the filesystem metadata would show
+inconsistent filesystem metadata because there would not appear to be any owner

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when

recovery finds
+an intent log item but does not find a corresponding intent done item, it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described in

the recovered

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy to

consider.
+The first is that log items must be added to a transaction in the correct order

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+to prevent conflicts with principal objects that are not held by the
+transaction.
+In other words, all per-AG metadata updates for an unmapped block

must be
+completed before the last update to free the extent, and extents should not

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers are

(usually)

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+released between each transaction in a chain.
+This means that other threads can observe an AG in an intermediate

state,
+but as long as the first subtlety is handled, this should not affect the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk, which

means that
+offline fsck never sees the temporary inconsistencies caused by deferred work

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+item processing.
+In this manner, XFS employs a form of eventual consistency to avoid

deadlocks

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink features,

it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single file

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other

file

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG per

transaction

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must be

done once to
+remove the space from a staging area and again to map it into the file!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To deal with this explosion in a calm manner, XFS expands its use of

deferred
+work items to cover most reverse mapping updates and all refcount updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree of eventual

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its

deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+However, online fsck changes the rules -- remember that although

physical
+updates to per-AG structures are coordinated by locking the buffers for AG

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data structure,

it must do

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock,

scrub may have

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a

reverse
+mapping update but not the corresponding refcount update, the two AG btrees
+will appear inconsistent to scrub and an observation of corruption will be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be

catastrophic!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Several solutions to this problem were evaluated upon discovery of
Quoted 1 line

this flaw:

Hmm, so while having a really in depth efi example is insightful, I wonder if it would be more oranized to put it in a separate document somewhere and just reference it. As far as ofsck is concerned, I think a lighter sumary would do:

"Complex operations that modify multiple AGs are performed through a series of transactions which are logged to a journal that an offline fsck can either replay or discard. Online fsck however, must be able to deal with these operations while they are still in progress. This presents a unique challenge for ofsck since a partially completed transaction chain may present the appearance of inconsistencies, even though the operations are functioning as intended. (For a more detailed example, see <cite document here...>)

Quoted 2 lines

The challenge then becomes how to avoid incorrectly repairing these non-issues as doing so would cause more harm than help."

I agree that this topic needs a much shorter introduction before moving on to the gory details. How does this strike you?

"Complex operations can make modifications to multiple per-AG data structures with a chain of transactions. These chains, once committed to the log, are restarted during log recovery if the system crashes while processing the chain. Because the AG header buffers are unlocked between transactions within a chain, online checking must coordinate with chained operations that are in progress to avoid incorrectly detecting inconsistencies due to pending chains. Furthermore, online repair must not run when operations are pending because the metadata are temporarily inconsistent with each other, and rebuilding is not possible."

"Only online fsck has this requirement of total consistency of AG metadata, and should be relatively rare as compared to filesystem change operations. Online fsck coordinates with transaction chains as follows:

* "For each AG, maintain a count of intent items targetting that AG. The count should be bumped whenever a new item is added to the chain. The count should be dropped when the filesystem has locked the AG header buffers and finished the work.

* "When online fsck wants to examine an AG, it should lock the AG header buffers to quiesce all transaction chains that want to modify that AG. If the count is zero, proceed with the checking operation. If it is nonzero, cycle the buffer locks to allow the chain to make forward progress.

"This may lead to online fsck taking a long time to complete, but regular filesystem updates take precedence over background checking activity. Details about the discovery of this situation are presented in the <next section>, and details about the solution are presented <after that>."

These gory details of how I recognized the problem are a subsection of the main heading, and anyone who wants to know them can read it. Readers who'd rather move on to the solution can jump directly to the "Intent Drains" section. The <bracketed> text are hyperlinks.

Quoted 20 lines
Quoted 20 lines

+
+1. Add a higher level lock to allocation groups and require writer threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it is
+   difficult to determine which locks need to be obtained, and in what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator

Quoted 8 lines
Quoted 7 lines

since it is
+   only loosely coupled with the actual deferred work items. +   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete before
+   work can start on a new sibling task.

Hmm, that one doesnt seem like it's really an option then :-(

Right. Now that this section has become its own gory details subsection, the sentence preceeding the numbered list becomes:

"Several other solutions to this problem were evaluated upon discovery of this flaw and rejected:"

Quoted 22 lines
Quoted 22 lines

+
+3. Teach online fsck to walk all transactions waiting for whichever lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG. +   When online fsck wants to examine an AG, it should lock the AG header
+   buffers to quiesce all transaction chains that want to modify that AG, and

Quoted 8 lines
Quoted 7 lines

+   only proceed with the scrub if the count is zero.
+   In other words, scrub only proceeds if it can lock the AG header buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.

So basically it sounds like 4 is the only reasonable option?

Yes.

Quoted 2 lines

If the discussion concerning the other options have died down, I would clean them out.

That's just the problem -- I've sent this and the code patches to the list several times now, and mostly haven't heard any solid replies. So it's a bit premature to take it out, and again it might be useful to capture the roads not taken.

Quoted 11 lines

They're great for brain storming and invitations for collaboration, but ideally the goal of any of that should be to narrow down an agreed upon plan of action. And the goal of your document should make clear what that plan is. So if no one has any objections by now, maybe just tie it right into the last line:

"The challenge then becomes how to avoid incorrectly repairing these non-issues as doing so would cause more harm than help. Fortunately only online fsck has this requirement of total consistency..."

Quoted 6 lines

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Intent Drains
+`````````````
+
+The fourth solution is implemented in the current iteration of

This solution is implemented...

"Online fsck uses an atomic intent item counter and lock cycling to coordinate with transaction chains. There are two key properties to the drain mechanism..."

Quoted 58 lines
Quoted 58 lines

online fsck,

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+with atomic_t providing the active intent counter.
+
+There are two key properties to the drain mechanism.
+First, the counter is incremented when a deferred work item is

*queued* to a
+transaction, and it is decremented after the associated intent done log item is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+*committed* to another transaction.
+The second property is that deferred work can be added to a

transaction without
+holding an AG header lock, but per-AG work items cannot be marked done without
+locking that AG header buffer to log the physical updates and the intent done

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+log item.
+The first property enables scrub to yield to running transaction

chains, which
+is an explicit deprioritization of online fsck to benefit file operations.
+The second property of the drain is key to the correct coordination of scrub,

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+since scrub will always be able to decide if a conflict is possible.
+
+For regular filesystem code, the drain works as follows:
+
+1. Call the appropriate subsystem function to add a deferred work

item to a
+   transaction.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+2. The function calls ``xfs_drain_bump`` to increase the counter.
+
+3. When the deferred item manager wants to finish the deferred work

item, it
+   calls ``->finish_item`` to complete it.
+
+4. The ``->finish_item`` implementation logs some changes and calls +   ``xfs_drain_drop`` to decrease the sloppy counter and wake up any threads
+   waiting on the drain.
+
+5. The subtransaction commits, which unlocks the resource associated with the
+   intent item.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+For scrub, the drain works as follows:
+
+1. Lock the resource(s) associated with the metadata being scrubbed.
+   For example, a scan of the refcount btree would lock the AGI and

AGF header
+   buffers.
+
+2. If the counter is zero (``xfs_drain_busy`` returns false), there are no
+   chains in progress and the operation may proceed.

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+3. Otherwise, release the resources grabbed in step 1.
+
Quoted 8 lines
Quoted 7 lines

+4. Wait for the intent counter to reach zero (``xfs_drain_intents``), then go
+   back to step 1 unless a signal has been caught.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To avoid polling in step 4, the drain provides a waitqueue for scrub

threads to
+be woken up whenever the intent count drops to zero.

I think all that makes sense

Good! :)

Quoted 113 lines
Quoted 113 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+The proposed patchset is the
+`scrub intent drain series
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-drain-intents>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +7 -0
+
+.. _jump_labels:
+
+Static Keys (aka Jump Label Patching)
+`````````````````````````````````````
+
+Online fsck for XFS separates the regular filesystem from the

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repair code as much as possible.
+However, there are a few parts of online fsck (such as the intent

drains, and
+later, live update hooks) where it is useful for the online fsck code to know

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+what's going on in the rest of the filesystem.
+Since it is not expected that online fsck will be constantly running

in the
+background, it is very important to minimize the runtime overhead imposed by
+these hooks when online fsck is compiled into the kernel but not actively

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running on behalf of userspace.
+Taking locks in the hot path of a writer thread to access a data

structure only
+to find that no further action is necessary is expensive -- on the author's

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer, this have an overhead of 40-50ns per access.
+Fortunately, the kernel supports dynamic code patching, which

enables XFS to
+replace a static branch to hook code with ``nop`` sleds when online fsck isn't

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running.
+This sled has an overhead of however long it takes the instruction

decoder to
+skip past the sled, which seems to be on the order of less than 1ns and

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+does not access memory outside of instruction fetching.
+
+When online fsck enables the static key, the sled is replaced with

an

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unconditional branch to call the hook code.
+The switchover is quite expensive (~22000ns) but is paid entirely by

the
+program that invoked online fsck, and can be amortized if multiple threads
+enter online fsck at the same time, or if multiple filesystems are being

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+checked at the same time.
+Changing the branch direction requires taking the CPU hotplug lock,

and since
+CPU initialization requires memory allocation, online fsck must be careful not
+to change a static key while holding any locks or resources that could be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+accessed in the memory reclaim paths.
+To minimize contention on the CPU hotplug lock, care should be taken

not to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+enable or disable static keys unnecessarily.
+
+Because static keys are intended to minimize hook overhead for

regular
+filesystem operations when xfs_scrub is not running, the intended usage

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+patterns are as follows:
+
+- The hooked part of XFS should declare a static-scoped static key

that
+  defaults to false.
+  The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this. +  The static key itself should be declared as a ``static`` variable. +
+- When deciding to invoke code that's only used by scrub, the regular
+  filesystem should call the ``static_branch_unlikely`` predicate to avoid the
+  scrub-only hook code if the static key is not enabled. +
+- The regular filesystem should export helper functions that call +  ``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
+  static key.
+  Wrapper functions make it easy to compile out the relevant code if the kernel
+  distributor turns off online fsck at build time.
+
+- Scrub functions wanting to turn on scrub-only XFS functionality should call
+  the ``xchk_fshooks_enable`` from the setup function to enable a specific
+  hook.
+  This must be done before obtaining any resources that are used by memory
+  reclaim.
+  Callers had better be sure they really need the functionality gated by the
+  static key; the ``TRY_HARDER`` flag is useful here.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Online scrub has resource acquisition helpers (e.g.

``xchk_perag_lock``) to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+handle locking AGI and AGF buffers for all scrubber functions.
+If it detects a conflict between scrub and the running transactions,

it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try to wait for intents to complete.
+If the caller of the helper has not enabled the static key, the

helper will
+return -EDEADLOCK, which should result in the scrub being restarted with the
+``TRY_HARDER`` flag set.

Quoted 8 lines
Quoted 5 lines

+The scrub setup function should detect that flag, enable the static key, and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try the scrub again.
+Scrub teardown disables all static keys obtained by

``xchk_fshooks_enable``.

Ok, this part here seems pretty well documented. Organizing nits aside I think it looks good.

Thanks for digging into all of this!

--D

Quoted 7 lines

Allison

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+For more information, please see the kernel documentation of
+Documentation/staging/static-keys.rst.
#31 lore raw

Re: [PATCH 06/14] xfs: document how online fsck deals with eventual consistency

Quoted 62 lines · Darrick J. Wong
Quoted 49 lines · Allison Henderson
Quoted 48 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Writes to an XFS filesystem employ an eventual consistency update model
to break up complex multistep metadata updates into small chained transactions.  This is generally good for performance and scalability
because XFS doesn't need to prepare for enormous transactions, but it
also means that online fsck must be careful not to attempt a fsck action
unless it can be shown that there are no other threads processing a
transaction chain.  This part of the design documentation covers the
thinking behind the consistency model and how scrub deals with it.

---
 .../filesystems/xfs-online-fsck-design.rst         |  303
++++++++++++++++++++

 1 file changed, 303 insertions(+)
Documentation/filesystems/xfs-online-fsck-design.rst +0 -0
index f45bf97fa9c4..419eb54ee200 100644
@@ -1443,3 +1443,306 @@ This step is critical for enabling system

administrator to monitor the status
 of the filesystem and the progress of any repairs. For developers, it is a useful means to judge the efficacy of error
detection
 and correction in the online and offline checking tools.

Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Eventual Consistency vs. Online Fsck
+------------------------------------
+
+Midway through the development of online scrubbing, the fsstress

tests
+uncovered a misinteraction between online fsck and compound transaction chains
+created by other writer threads that resulted in false reports of
metadata

Quoted 11 lines
Quoted 8 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+inconsistency.
+The root cause of these reports is the eventual consistency

model
introduced by
+the expansion of deferred work items and compound transaction chains
when
+reverse mapping and reflink were introduced.

Quoted 1 line

Was there supposed to be a comment here?

No, sometimes I'll fiddle with paraphrasing, but if it's not enough of an improvement, I'll scrap it, but I think evolution leaves the white space

Quoted 296 lines
Quoted 247 lines
Quoted 220 lines
Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Originally, transaction chains were added to XFS to avoid

deadlocks
when

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unmapping space from files.
+Deadlock avoidance rules require that AGs only be locked in

increasing order,
+which makes it impossible (say) to use a single transaction to free
a space
+extent in AG 7 and then try to free a now superfluous block mapping
btree block

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+in AG 3.
+To avoid these kinds of deadlocks, XFS creates Extent Freeing

Intent
(EFI) log
+items to commit to freeing some space in one transaction while deferring the

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+actual metadata updates to a fresh transaction.
+The transaction sequence looks like this:
+
+1. The first transaction contains a physical update to the

file's
block mapping
+   structures to remove the mapping from the btree blocks. +   It then attaches to the in-memory transaction an action item to
schedule
+   deferred freeing of space.
+   Concretely, each transaction maintains a list of ``struct +   xfs_defer_pending`` objects, each of which maintains a list of
``struct
+   xfs_extent_free_item`` objects.
+   Returning to the example above, the action item tracks the freeing of both
+   the unmapped space from AG 7 and the block mapping btree (BMBT)
block from
+   AG 3.
+   Deferred frees recorded in this manner are committed in the log
by creating
+   an EFI log item from the ``struct xfs_extent_free_item`` object
and
+   attaching the log item to the transaction.
+   When the log is persisted to disk, the EFI item is written into
the ondisk
+   transaction record.
+   EFIs can list up to 16 extents to free, all sorted in AG order.
+
+2. The second transaction contains a physical update to the free space btrees
+   of AG 3 to release the former BMBT block and a second physical
update to the
+   free space btrees of AG 7 to release the unmapped file space. +   Observe that the the physical updates are resequenced in the correct order
+   when possible.
+   Attached to the transaction is a an extent free done (EFD) log
item.
+   The EFD contains a pointer to the EFI logged in transaction #1 so
that log
+   recovery can tell if the EFI needs to be replayed.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+If the system goes down after transaction #1 is written back to

the
filesystem
+but before #2 is committed, a scan of the filesystem metadata would
show
+inconsistent filesystem metadata because there would not appear to
be any owner

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+of the unmapped space.
+Happily, log recovery corrects this inconsistency for us -- when

recovery finds
+an intent log item but does not find a corresponding intent done item, it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+reconstruct the incore state of the intent item and finish it.
+In the example above, the log must replay both frees described

in
the recovered

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+EFI to complete the recovery phase.
+
+There are two subtleties to XFS' transaction chaining strategy

to
consider.
+The first is that log items must be added to a transaction in the
correct order
+to prevent conflicts with principal objects that are not held by the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+transaction.
+In other words, all per-AG metadata updates for an unmapped

block
must be
+completed before the last update to free the extent, and extents should not

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+be reallocated until that last update commits to the log.
+The second subtlety comes from the fact that AG header buffers

are
(usually)

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+released between each transaction in a chain.
+This means that other threads can observe an AG in an

intermediate
state,
+but as long as the first subtlety is handled, this should not affect
the

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+correctness of filesystem operations.
+Unmounting the filesystem flushes all pending work to disk,

which
means that
+offline fsck never sees the temporary inconsistencies caused by deferred work

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+item processing.
+In this manner, XFS employs a form of eventual consistency to

avoid
deadlocks

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+and increase parallelism.
+
+During the design phase of the reverse mapping and reflink

features,
it was
+decided that it was impractical to cram all the reverse mapping updates for a
+single filesystem change into a single transaction because a single
file

Documentation/filesystems/xfs-online-fsck-design.rst +18 -0
+mapping operation can explode into many small updates:
+
+* The block mapping update itself
+* A reverse mapping update for the block mapping update
+* Fixing the freelist
+* A reverse mapping update for the freelist fix
+
+* A shape change to the block mapping btree
+* A reverse mapping update for the btree update
+* Fixing the freelist (again)
+* A reverse mapping update for the freelist fix
+
+* An update to the reference counting information
+* A reverse mapping update for the refcount update
+* Fixing the freelist (a third time)
+* A reverse mapping update for the freelist fix
+
+* Freeing any space that was unmapped and not owned by any other

file

Documentation/filesystems/xfs-online-fsck-design.rst +8 -0
+* Fixing the freelist (a fourth time)
+* A reverse mapping update for the freelist fix
+
+* Freeing the space used by the block mapping btree
+* Fixing the freelist (a fifth time)
+* A reverse mapping update for the freelist fix
+
+Free list fixups are not usually needed more than once per AG

per
transaction

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+chain, but it is theoretically possible if space is very tight.
+For copy-on-write updates this is even worse, because this must

be
done once to
+remove the space from a staging area and again to map it into the
file!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To deal with this explosion in a calm manner, XFS expands its

use of
deferred
+work items to cover most reverse mapping updates and all refcount
updates.
+This reduces the worst case size of transaction reservations by breaking the
+work into a long chain of small updates, which increases the degree
of eventual

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+consistency in the system.
+Again, this generally isn't a problem because XFS orders its

deferred work
+items carefully to avoid resource reuse conflicts between unsuspecting threads.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+However, online fsck changes the rules -- remember that although

physical
+updates to per-AG structures are coordinated by locking the buffers
for AG

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+headers, buffer locks are dropped between transactions.
+Once scrub acquires resources and takes locks for a data

structure,
it must do

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+all the validation work without releasing the lock.
+If the main lock for a space btree is an AG header buffer lock,

scrub may have
+interrupted another thread that is midway through finishing a chain.
+For example, if a thread performing a copy-on-write has completed a
reverse
+mapping update but not the corresponding refcount update, the two AG
btrees
+will appear inconsistent to scrub and an observation of corruption
will be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+recorded.  This observation will not be correct.
+If a repair is attempted in this state, the results will be

catastrophic!

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Several solutions to this problem were evaluated upon discovery
Quoted 2 lines

of
this flaw:

Hmm, so while having a really in depth efi example is insightful, I wonder if it would be more oranized to put it in a separate document
somewhere and just reference it.  As far as ofsck is concerned, I think
a lighter sumary would do:

"Complex operations that modify multiple AGs are performed through a
series of transactions which are logged to a journal that an offline
fsck can either replay or discard.  Online fsck however, must be able
to deal with these operations while they are still in progress. This
presents a unique challenge for ofsck since a partially completed transaction chain may present the appearance of inconsistencies, even
though the operations are functioning as intended. (For a more detailed
example, see <cite document here...>)

Quoted 2 lines

The challenge then becomes how to avoid incorrectly repairing these non-issues as doing so would cause more harm than help."

I agree that this topic needs a much shorter introduction before moving
on to the gory details.  How does this strike you?

"Complex operations can make modifications to multiple per-AG data structures with a chain of transactions.  These chains, once committed
to the log, are restarted during log recovery if the system crashes while processing the chain.  Because the AG header buffers are unlocked
between transactions within a chain, online checking must coordinate with chained operations that are in progress to avoid incorrectly detecting inconsistencies due to pending chains.  Furthermore, online repair must not run when operations are pending because the metadata are
temporarily inconsistent with each other, and rebuilding is not possible."

"Only online fsck has this requirement of total consistency of AG metadata, and should be relatively rare as compared to filesystem change
operations.  Online fsck coordinates with transaction chains as follows:

* "For each AG, maintain a count of intent items targetting that AG. The count should be bumped whenever a new item is added to the chain.
  The count should be dropped when the filesystem has locked the AG header buffers and finished the work.

* "When online fsck wants to examine an AG, it should lock the AG header
  buffers to quiesce all transaction chains that want to modify that AG.
  If the count is zero, proceed with the checking operation.  If it is
  nonzero, cycle the buffer locks to allow the chain to make forward progress.

"This may lead to online fsck taking a long time to complete, but regular filesystem updates take precedence over background checking activity.  Details about the discovery of this situation are presented
in the <next section>, and details about the solution are presented <after that>."

Quoted 5 lines

These gory details of how I recognized the problem are a subsection of
the main heading, and anyone who wants to know them can read it. Readers who'd rather move on to the solution can jump directly to the "Intent Drains" section.  The <bracketed> text are hyperlinks.

Ok, I think that works. Much lighter, and more to the point for ofsck

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines

+
+1. Add a higher level lock to allocation groups and require writer
threads to
+   acquire the higher level lock in AG order before making any changes.
+   This would be very difficult to implement in practice because it
is
+   difficult to determine which locks need to be obtained, and in
what order,
+   without simulating the entire operation.
+   Performing a dry run of a file operation to discover necessary
locks would
+   make the filesystem very slow.
+
+2. Make the deferred work coordinator code aware of consecutive intent items
+   targeting the same AG and have it hold the AG header buffers locked across
+   the transaction roll between updates.
+   This would introduce a lot of complexity into the coordinator since it is

Quoted 8 lines
Quoted 7 lines

+   only loosely coupled with the actual deferred work items. +   It would also fail to solve the problem because deferred work items can
+   generate new deferred subtasks, but all subtasks must be complete
before
+   work can start on a new sibling task.

Hmm, that one doesnt seem like it's really an option then :-(

Quoted 6 lines

Right.  Now that this section has become its own gory details subsection, the sentence preceeding the numbered list becomes:

"Several other solutions to this problem were evaluated upon discovery
of this flaw and rejected:"

Ok

Quoted 71 lines
Quoted 29 lines
Quoted 29 lines

+
+3. Teach online fsck to walk all transactions waiting for whichever
lock(s)
+   protect the data structure being scrubbed to look for pending operations.
+   The checking and repair operations must factor these pending operations into
+   the evaluations being performed.
+   This solution is a nonstarter because it is *extremely* invasive
to the main
+   filesystem.
+
+4. Recognize that only online fsck has this requirement of total consistency
+   of AG metadata, and that online fsck should be relatively rare as
compared
+   to filesystem change operations.
+   For each AG, maintain a count of intent items targetting that AG.
+   When online fsck wants to examine an AG, it should lock the AG
header
+   buffers to quiesce all transaction chains that want to modify that AG, and
+   only proceed with the scrub if the count is zero.

Quoted 8 lines
Quoted 7 lines

+   In other words, scrub only proceeds if it can lock the AG header
buffers and
+   there can't possibly be any intents in progress.
+   This may lead to fairness and starvation issues, but regular filesystem
+   updates take precedence over online fsck activity.

So basically it sounds like 4 is the only reasonable option?

Yes.

Quoted 3 lines

If the discussion concerning the other options have died down, I would
clean them out.

That's just the problem -- I've sent this and the code patches to the list several times now, and mostly haven't heard any solid replies. So
it's a bit premature to take it out, and again it might be useful to capture the roads not taken.

Quoted 14 lines

They're great for brain storming and invitations for collaboration, but ideally the goal of any of that should be to narrow
down an agreed upon plan of action.  And the goal of your document should make clear what that plan is.  So if no one has any objections
by now, maybe just tie it right into the last line:

"The challenge then becomes how to avoid incorrectly repairing these
non-issues as doing so would cause more harm than help. Fortunately only online fsck has this requirement of total consistency..."

Quoted 6 lines

Documentation/filesystems/xfs-online-fsck-design.rst This step is critical for enabling system administrator to monitor the status

Quoted 5 lines
Documentation/filesystems/xfs-online-fsck-design.rst +5 -0
+
+Intent Drains
+`````````````
+
+The fourth solution is implemented in the current iteration of

This solution is implemented...

Quoted 4 lines

"Online fsck uses an atomic intent item counter and lock cycling to coordinate with transaction chains.  There are two key properties to the
drain mechanism..."

Ok, sounds fine

Quoted 245 lines
Quoted 74 lines
Quoted 74 lines

online fsck,

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+with atomic_t providing the active intent counter.
+
+There are two key properties to the drain mechanism.
+First, the counter is incremented when a deferred work item is

*queued* to a
+transaction, and it is decremented after the associated intent done
log item is

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+*committed* to another transaction.
+The second property is that deferred work can be added to a

transaction without
+holding an AG header lock, but per-AG work items cannot be marked
done without
+locking that AG header buffer to log the physical updates and the
intent done

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+log item.
+The first property enables scrub to yield to running transaction

chains, which
+is an explicit deprioritization of online fsck to benefit file operations.
+The second property of the drain is key to the correct coordination
of scrub,
+since scrub will always be able to decide if a conflict is possible.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+For regular filesystem code, the drain works as follows:
+
+1. Call the appropriate subsystem function to add a deferred

work
item to a
+   transaction.
+
+2. The function calls ``xfs_drain_bump`` to increase the counter.
+
+3. When the deferred item manager wants to finish the deferred work
item, it
+   calls ``->finish_item`` to complete it.
+
+4. The ``->finish_item`` implementation logs some changes and calls
+   ``xfs_drain_drop`` to decrease the sloppy counter and wake up any
threads
+   waiting on the drain.
+
+5. The subtransaction commits, which unlocks the resource associated
with the
+   intent item.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+For scrub, the drain works as follows:
+
+1. Lock the resource(s) associated with the metadata being

scrubbed.
+   For example, a scan of the refcount btree would lock the AGI and
AGF header
+   buffers.
+
+2. If the counter is zero (``xfs_drain_busy`` returns false), there
are no
+   chains in progress and the operation may proceed.

Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+3. Otherwise, release the resources grabbed in step 1.
+
+4. Wait for the intent counter to reach zero
Quoted 8 lines
Quoted 7 lines

(``xfs_drain_intents``), then go
+   back to step 1 unless a signal has been caught.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+To avoid polling in step 4, the drain provides a waitqueue for

scrub
threads to
+be woken up whenever the intent count drops to zero.

I think all that makes sense

Good! :)

Quoted 140 lines
Quoted 140 lines
Documentation/filesystems/xfs-online-fsck-design.rst +4 -0
+
+The proposed patchset is the
+`scrub intent drain series
+<

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/
log/?h=scrub-drain-intents>`_.

Documentation/filesystems/xfs-online-fsck-design.rst +7 -0
+
+.. _jump_labels:
+
+Static Keys (aka Jump Label Patching)
+`````````````````````````````````````
+
+Online fsck for XFS separates the regular filesystem from the

checking and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+repair code as much as possible.
+However, there are a few parts of online fsck (such as the

intent
drains, and
+later, live update hooks) where it is useful for the online fsck code to know

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+what's going on in the rest of the filesystem.
+Since it is not expected that online fsck will be constantly

running
in the
+background, it is very important to minimize the runtime overhead
imposed by
+these hooks when online fsck is compiled into the kernel but not actively

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running on behalf of userspace.
+Taking locks in the hot path of a writer thread to access a data

structure only
+to find that no further action is necessary is expensive -- on the
author's

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+computer, this have an overhead of 40-50ns per access.
+Fortunately, the kernel supports dynamic code patching, which

enables XFS to
+replace a static branch to hook code with ``nop`` sleds when online
fsck isn't

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+running.
+This sled has an overhead of however long it takes the

instruction
decoder to
+skip past the sled, which seems to be on the order of less than 1ns
and

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+does not access memory outside of instruction fetching.
+
+When online fsck enables the static key, the sled is replaced

with
an

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+unconditional branch to call the hook code.
+The switchover is quite expensive (~22000ns) but is paid

entirely by
the
+program that invoked online fsck, and can be amortized if multiple
threads
+enter online fsck at the same time, or if multiple filesystems are
being

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+checked at the same time.
+Changing the branch direction requires taking the CPU hotplug

lock,
and since
+CPU initialization requires memory allocation, online fsck must be
careful not
+to change a static key while holding any locks or resources that could be

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+accessed in the memory reclaim paths.
+To minimize contention on the CPU hotplug lock, care should be

taken
not to

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+enable or disable static keys unnecessarily.
+
+Because static keys are intended to minimize hook overhead for

regular
+filesystem operations when xfs_scrub is not running, the intended
usage

Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+patterns are as follows:
+
+- The hooked part of XFS should declare a static-scoped static

key
that
+  defaults to false.
+  The ``DEFINE_STATIC_KEY_FALSE`` macro takes care of this. +  The static key itself should be declared as a ``static`` variable.
+
+- When deciding to invoke code that's only used by scrub, the regular
+  filesystem should call the ``static_branch_unlikely`` predicate to
avoid the
+  scrub-only hook code if the static key is not enabled. +
+- The regular filesystem should export helper functions that call
+  ``static_branch_inc`` to enable and ``static_branch_dec`` to disable the
+  static key.
+  Wrapper functions make it easy to compile out the relevant code if
the kernel
+  distributor turns off online fsck at build time.
+
+- Scrub functions wanting to turn on scrub-only XFS functionality
should call
+  the ``xchk_fshooks_enable`` from the setup function to enable a
specific
+  hook.
+  This must be done before obtaining any resources that are used by
memory
+  reclaim.
+  Callers had better be sure they really need the functionality gated by the
+  static key; the ``TRY_HARDER`` flag is useful here.

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+
+Online scrub has resource acquisition helpers (e.g.

``xchk_perag_lock``) to

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+handle locking AGI and AGF buffers for all scrubber functions.
+If it detects a conflict between scrub and the running

transactions,
it will

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try to wait for intents to complete.
+If the caller of the helper has not enabled the static key, the

helper will
+return -EDEADLOCK, which should result in the scrub being restarted
with the
+``TRY_HARDER`` flag set.
+The scrub setup function should detect that flag, enable the static

Quoted 8 lines
Quoted 4 lines

key, and

Documentation/filesystems/xfs-online-fsck-design.rst +2 -0
+try the scrub again.
+Scrub teardown disables all static keys obtained by

``xchk_fshooks_enable``.

Ok, this part here seems pretty well documented.  Organizing nits aside
I think it looks good.

Thanks for digging into all of this!

--D

Quoted 7 lines

Allison

Quoted 4 lines
Documentation/filesystems/xfs-online-fsck-design.rst +3 -0
+
+For more information, please see the kernel documentation of
+Documentation/staging/static-keys.rst.