Skip to contentSkip to conversation

xfs-docs: metadata directories and realtime groups

PATCHBOMB 6.13 v5.5 New Darrick J. Wong 2 participants 20 messages Nov 6, 2024 – Nov 7, 2024

lore
Share & tools
lore
#0 lore raw

Hi everyone,

Now that I've sent off the metadir PR, here are the relevant changes to the ondisk format documentation for the last few bits of online fsck and the metadata directory tree / realtime allocation groups feature.

vNobody likes asciidoc, so if you want an easy to read version, try either:
http://djwong.org/docs/xfs_filesystem_structure.pdf
http://djwong.org/docs/xfs_filesystem_structure.html

Please have a look at the git tree links for code changes:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-groups_2024-11-06

--D

#1 lore raw

PATCH 1/3 xfs-documentation: last few updates for online fsck

Hi all,

Now that we've landed online fsck, let's update the chapter on metadata reconstruction is supposed to work, and document the new filesystem property that we use to control it.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

Comments and questions are, as always, welcome.

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=online-fsck
---
Commits in this patchset:
* design: update metadata reconstruction chapter
* design: document filesystem properties

---
 .../fs_properties.asciidoc                         |   28 ++++++++++++++++++++
 .../reconstruction.asciidoc                        |   17 ++++++------
 .../xfs_filesystem_structure.asciidoc              |    2 +
 3 files changed, 39 insertions(+), 8 deletions(-)
 create mode 100644 design/XFS_Filesystem_Structure/fs_properties.asciidoc
#2 lore raw

PATCH 1/2 design: update metadata reconstruction chapter

From: Darrick J. Wong <[email protected]>

We've landed online repair and full backrefs in the filesystem, so update the links to the new sections and transform future tense to present tense.

---
 .../reconstruction.asciidoc                        |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
design/XFS_Filesystem_Structure/reconstruction.asciidoc +9 -8
index f172e0f8161656..f4c10217910b6c 100644
@@ -1,10 +1,6 @@
 [[Reconstruction]]
 = Metadata Reconstruction
 
-[NOTE]
-This is a theoretical discussion of how reconstruction could work; none of this
-is implemented as of 2015.
-
 A simple UNIX filesystem can be thought of in terms of a directed acyclic graph.
 To a first approximation, there exists a root directory node, which points to
 other nodes.  Those other nodes can themselves be directories or they can be
@@ -45,9 +41,14 @@ The xref:Reverse_Mapping_Btree[reverse-mapping B+tree] fills in part of the
 puzzle.  Since it contains copies of every entry in each inode’s data and
 attribute forks, we can fix a corrupted block map with these records.
 Furthermore, if the inode B+trees become corrupt, it is possible to visit all
-inode chunks using the reverse-mapping data.  Should XFS ever gain the ability
-to store parent directory information in each inode, it also becomes possible
+inode chunks using the reverse-mapping data.  xref:Parent_Pointers[Directory
+parent pointers] fill in the rest of the puzzle by mirroring the directory tree
+structure with parent directory information in each inode.  It is now possible
 to resurrect damaged directory trees, which should reduce the complaints about
 inodes ending up in +/lost+found+.  Everything else in the per-AG primary
-metadata can already be reconstructed via +xfs_repair+.  Hopefully,
-reconstruction will not turn out to be a fool's errand.
+metadata can already be reconstructed via +xfs_repair+.
+
+See the
+https://docs.kernel.org/filesystems/xfs/xfs-online-fsck-design.html[design
+document] for online repair for a more thorough discussion of how this metadata
+are put to use.
#4 lore raw

PATCH 2/2 design: document filesystem properties

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Now that xfsprogs utilities can set properties to coordinate the behavior of other xfsprogs utilities, record them in the ondisk format documentation.

---
 .../fs_properties.asciidoc                         |   28 ++++++++++++++++++++
 .../xfs_filesystem_structure.asciidoc              |    2 +
 2 files changed, 30 insertions(+)
 create mode 100644 design/XFS_Filesystem_Structure/fs_properties.asciidoc
design/XFS_Filesystem_Structure/fs_properties.asciidoc new file +28 -0
new file mode 100644
index 00000000000000..b639aec9ab6366
@@ -0,0 +1,28 @@
+[[Filesystem_Properties]]
+= Filesystem Properties
+
+System administrators can set filesystem-wide properties to coordinate the
+behavior of userspace XFS administration tools.  These properties are recorded
+as extended attributes of the +ATTR_ROOT+ namesace that are set on the root
+directory.
+
+[options="header"]
+|=====
+| Property			| Description
+| +xfs:autofsck+		| Online fsck background scanning behavior
+|=====
+
+*xfs:autofsck*::
+This property controls the behavior of background online fsck.
+Unrecognized values are treated as if the property was not set.
+Check the +xfs_scrub+ manual page for more information.
+
+.autofsck property values
+[options="header"]
+|=====
+| Value				| Description
+| +none+			| Do not perform background scans.
+| +check+			| Only check metadata.
+| +optimize+			| Check and optimize metadata.
+| +repair+			| Check, repair, or optimize metadata.
+|=====
design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc +2 -0
index a95a5806172a0c..689e2a874c13e9 100644
@@ -84,6 +84,8 @@ include::journaling_log.asciidoc[]
 
 include::internal_inodes.asciidoc[]
 
+include::fs_properties.asciidoc[]
+
 :leveloffset: 0
 
 Dynamically Allocated Structures
#6 lore raw

PATCH 2/3 xfs-documentation: document metadata directories

In reply to Darrick J. Wong

Hi all,

This patch documents the metadata directory tree feature.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

Comments and questions are, as always, welcome.

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=metadir

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=metadir

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=metadir

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=metadir
---
Commits in this patchset:
* design: document the changes required to handle metadata directories

---
 .../allocation_groups.asciidoc                     |   14 +-
 .../internal_inodes.asciidoc                       |  113 ++++++++++++++++++++
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |   22 ++++
 3 files changed, 142 insertions(+), 7 deletions(-)
#7 lore raw

PATCH 1/1 design: document the changes required to handle metadata directories

From: Darrick J. Wong <[email protected]>

Document the ondisk format changes for metadata directories.

---
 .../allocation_groups.asciidoc                     |   14 +-
 .../internal_inodes.asciidoc                       |  113 ++++++++++++++++++++
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |   22 ++++
 3 files changed, 142 insertions(+), 7 deletions(-)
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +8 -6
index d7fd63ea20a646..ec59519dc2ffc1 100644
@@ -105,7 +105,7 @@ struct xfs_sb
 	xfs_ino_t		sb_pquotino;
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
-	xfs_ino_t		sb_rrmapino;
+	xfs_ino_t		sb_metadirino;
 };
 ----
 *sb_magicnum*::
@@ -472,6 +472,10 @@ information.
 Directory parent pointers.  See the section about xref:Parent_Pointers[parent
 pointers] for more information.
 
+| +XFS_SB_FEAT_INCOMPAT_METADIR+ |
+Metadata directory tree.  See the section about the xref:Metadata_Directories[
+metadata directory tree] for more information.
+
 |=====
 
 *sb_features_log_incompat*::
@@ -505,11 +509,9 @@ If the +XFS_SB_FEAT_INCOMPAT_META_UUID+ feature is set, then the UUID field in
 all metadata blocks must match this UUID.  If not, the block header UUID field
 must match +sb_uuid+.
 
-*sb_rrmapino*::
-If the +XFS_SB_FEAT_RO_COMPAT_RMAPBT+ feature is set and a real-time
-device is present (+sb_rblocks+ > 0), this field points to an inode
-that contains the root to the
-xref:Real_time_Reverse_Mapping_Btree[Real-Time Reverse Mapping B+tree].
+*sb_metadirino*::
+If the +XFS_SB_FEAT_RO_INCOMPAT_METADIR+ feature is set, this field points to
+the inode of the root directory of the metadata directory tree.
 This field is zero otherwise.
 
 === xfs_db Superblock Example
design/XFS_Filesystem_Structure/internal_inodes.asciidoc +113 -0

119 lines not on this page · raw message

design/XFS_Filesystem_Structure/ondisk_inode.asciidoc +21 -1
index 34c064871cb255..02ec0d12bb57e5 100644
@@ -78,7 +78,10 @@ struct xfs_dinode_core {
      __uint16_t                di_mode;
      __int8_t                  di_version;
      __int8_t                  di_format;
-     __uint16_t                di_onlink;
+     union {
+          __uint16_t           di_onlink;
+          __uint16_t           di_metatype;
+     };
      __uint32_t                di_uid;
      __uint32_t                di_gid;
      __uint32_t                di_nlink;
@@ -188,6 +191,17 @@ In v1 inodes, this specifies the number of links to the inode from directories.
 When the number exceeds 65535, the inode is converted to v2 and the link count
 is stored in +di_nlink+.
 
+*di_metatype*::
+If the +XFS_SB_FEAT_INCOMPAT_METADIR+ feature is enabled, the +di_onlink+ field
+is redefined to declare the intended contents of files in the metadata
+directory tree.
+
+[source, c]
+----
+enum xfs_metafile_type {
+};
+----
+
 *di_uid*::
 Specifies the owner's UID of the inode.
 
@@ -383,6 +397,12 @@ will be copied to all newly created files and directories.
 Files with this flag set may have up to (2^48^ - 1) extents mapped to the data
 fork and up to (2^32^ - 1) extents mapped to the attribute fork.  This flag
 requires the +XFS_SB_FEAT_INCOMPAT_NREXT64+ feature to be enabled.
+| +XFS_DIFLAG2_METADATA+	|
+This file contains filesystem metadata.  This feature requires the
++XFS_SB_FEAT_INCOMPAT_METADIR+ feature to be enabled.  See the section about
+xref:Metadata_Directories[metadata directories] for more information on
+metadata inode properties.  Only directories and regular files can have this
+flag set.
 |=====
 
 *di_cowextsize*::
Reviewed-by on v1 1/1
#8 lore raw

Re: [PATCH 1/1] design: document the changes required to handle metadata directories

Quoted 17 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Document the ondisk format changes for metadata directories.

---
 .../allocation_groups.asciidoc                     |   14 +-
 .../internal_inodes.asciidoc                       |  113 ++++++++++++++++++++
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |   22 ++++
 3 files changed, 142 insertions(+), 7 deletions(-)
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +0 -0
index d7fd63ea20a646..ec59519dc2ffc1 100644

design/XFS_Filesystem_Structure/allocation_groups.asciidoc struct xfs_sb · line 109

Quoted 6 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +1 -1
@@ -105,7 +105,7 @@ struct xfs_sb
 	xfs_ino_t		sb_pquotino;
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
-	xfs_ino_t		sb_rrmapino;
+	xfs_ino_t		sb_metadirino;
 };

Not new here, but I find it a bit odd that the super block is documented in allocation_groups.asciidoc.

The change itself looks good:

#9 lore raw

Re: [PATCH 1/1] design: document the changes required to handle metadata directories

Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Document the ondisk format changes for metadata directories.

---
 .../allocation_groups.asciidoc                     |   14 +-
 .../internal_inodes.asciidoc                       |  113 ++++++++++++++++++++
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |   22 ++++
 3 files changed, 142 insertions(+), 7 deletions(-)
Quoted 14 lines

design/XFS_Filesystem_Structure/allocation_groups.asciidoc struct xfs_sb · line 109

Quoted 11 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +1 -1
index d7fd63ea20a646..ec59519dc2ffc1 100644
@@ -105,7 +105,7 @@ struct xfs_sb
 	xfs_ino_t		sb_pquotino;
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
-	xfs_ino_t		sb_rrmapino;
+	xfs_ino_t		sb_metadirino;
 };

Not new here, but I find it a bit odd that the super block is documented in allocation_groups.asciidoc.

Me too. superblock.asciidoc?

Quoted 3 lines

The change itself looks good:

Thanks!

--D

#10 lore raw

PATCH 3/3 xfs-documentation: shard the realtime section

In reply to Darrick J. Wong

Hi all,

Document changes to the ondisk format for realtime groups.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

Comments and questions are, as always, welcome.

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-groups

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-groups

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-groups

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-groups
---
Commits in this patchset:
* design: move discussion of realtime volumes to a separate section
* design: document realtime groups
* design: document metadata directory tree quota changes
* design: update metadump v2 format to reflect rt dumps

---
 .../allocation_groups.asciidoc                     |   54 ++-
 .../XFS_Filesystem_Structure/common_types.asciidoc |    4 
 .../internal_inodes.asciidoc                       |   41 --
 design/XFS_Filesystem_Structure/magic.asciidoc     |    3 
 design/XFS_Filesystem_Structure/metadump.asciidoc  |   12 +
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |    5 
 design/XFS_Filesystem_Structure/realtime.asciidoc  |  394 ++++++++++++++++++++
 .../xfs_filesystem_structure.asciidoc              |    2 
 8 files changed, 459 insertions(+), 56 deletions(-)
 create mode 100644 design/XFS_Filesystem_Structure/realtime.asciidoc
#11 lore raw

PATCH 1/4 design: move discussion of realtime volumes to a separate section

From: Darrick J. Wong <[email protected]>

In preparation for documenting the realtime modernization project, move the discussions of the realtime-realted ondisk metadata to a separate file. Since realtime reverse mapping btrees haven't been added to the filesystem yet, stop including them in the final output.

---
 .../allocation_groups.asciidoc                     |   20 --------
 .../internal_inodes.asciidoc                       |   36 +-------------
 design/XFS_Filesystem_Structure/realtime.asciidoc  |   50 ++++++++++++++++++++
 .../xfs_filesystem_structure.asciidoc              |    2 +
 4 files changed, 54 insertions(+), 54 deletions(-)
 create mode 100644 design/XFS_Filesystem_Structure/realtime.asciidoc
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +0 -20
index ec59519dc2ffc1..5d560073d146ff 100644
@@ -1322,23 +1322,3 @@ core.magic = 0x494e
 
 The chunk record also indicates that this chunk has 32 inodes, and that the
 missing inodes are also ``free''.
-
-[[Real-time_Devices]]
-== Real-time Devices
-
-The performance of the standard XFS allocator varies depending on the internal
-state of the various metadata indices enabled on the filesystem.  For
-applications which need to minimize the jitter of allocation latency, XFS
-supports the notion of a ``real-time device''.  This is a special device
-separate from the regular filesystem where extent allocations are tracked with
-a bitmap and free space is indexed with a two-dimensional array.  If an inode
-is flagged with +XFS_DIFLAG_REALTIME+, its data will live on the real time
-device.  The metadata for real time devices is discussed in the section about
-xref:Real-time_Inodes[real time inodes].
-
-By placing the real time device (and the journal) on separate high-performance
-storage devices, it is possible to reduce most of the unpredictability in I/O
-response times that come from metadata operations.
-
-None of the XFS per-AG B+trees are involved with real time files.  It is not
-possible for real time files to share data blocks.
design/XFS_Filesystem_Structure/internal_inodes.asciidoc +2 -34
index eaa0a50aa848f3..68c86d30ff8206 100644
@@ -287,41 +287,9 @@ Log sequence number of the last DQ block write.
 *dd_crc*::
 Checksum of the DQ block.
 
-
 [[Real-time_Inodes]]
 == Real-time Inodes
 
 There are two inodes allocated to managing the real-time device's space, the
-Bitmap Inode and the Summary Inode.
-
-[[Real-Time_Bitmap_Inode]]
-=== Real-Time Bitmap Inode
-
-The real time bitmap inode, +sb_rbmino+, tracks the used/free space in the
-real-time device using an old-style bitmap. One bit is allocated per real-time
-extent. The size of an extent is specified by the superblock's +sb_rextsize+
-value.
-
-The number of blocks used by the bitmap inode is equal to the number of
-real-time extents (+sb_rextents+) divided by the block size (+sb_blocksize+)
-and bits per byte. This value is stored in +sb_rbmblocks+. The nblocks and
-extent array for the inode should match this.  Each real time block gets its
-own bit in the bitmap.
-
-[[Real-Time_Summary_Inode]]
-=== Real-Time Summary Inode
-
-The real time summary inode, +sb_rsumino+, tracks the used and free space
-accounting information for the real-time device.  This file indexes the
-approximate location of each free extent on the real-time device first by
-log2(extent size) and then by the real-time bitmap block number.  The size of
-the summary inode file is equal to +sb_rbmblocks+ × log2(realtime device size)
-× sizeof(+xfs_suminfo_t+).  The entry for a given log2(extent size) and
-rtbitmap block number is 0 if there is no free extents of that size at that
-rtbitmap location, and positive if there are any.
-
-This data structure is not particularly space efficient, however it is a very
-fast way to provide the same data as the two free space B+trees for regular
-files since the space is preallocated and metadata maintenance is minimal.
-
-include::rtrmapbt.asciidoc[]
+xref:Real-Time_Bitmap_Inode[Bitmap Inode] and the
+xref:Real-Time_Summary_Inode[Summary Inode].
design/XFS_Filesystem_Structure/realtime.asciidoc new file +50 -0

50 lines not on this page · raw message

design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc +2 -0
index 689e2a874c13e9..a643d18add6094 100644
@@ -84,6 +84,8 @@ include::journaling_log.asciidoc[]
 
 include::internal_inodes.asciidoc[]
 
+include::realtime.asciidoc[]
+
 include::fs_properties.asciidoc[]
 
 :leveloffset: 0
Reviewed-by on v1 1/4
#12 lore raw

Re: [PATCH 1/4] design: move discussion of realtime volumes to a separate section

Quoted 6 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In preparation for documenting the realtime modernization project, move the discussions of the realtime-realted ondisk metadata to a separate file. Since realtime reverse mapping btrees haven't been added to the filesystem yet, stop including them in the final output.

Looks good:

#13 lore raw

PATCH 2/4 design: document realtime groups

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Document the ondisk changes for realtime allocation groups.

---
 .../allocation_groups.asciidoc                     |   31 ++
 .../XFS_Filesystem_Structure/common_types.asciidoc |    4 
 .../internal_inodes.asciidoc                       |    2 
 design/XFS_Filesystem_Structure/magic.asciidoc     |    3 
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |    2 
 design/XFS_Filesystem_Structure/realtime.asciidoc  |  344 ++++++++++++++++++++
 6 files changed, 385 insertions(+), 1 deletion(-)
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +31 -0
index 5d560073d146ff..9f92be49a7a095 100644
@@ -106,6 +106,10 @@ struct xfs_sb
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
 	xfs_ino_t		sb_metadirino;
+	xfs_rgnumber_t		sb_rgcount;
+	xfs_rgblock_t		sb_rgextents;
+	uint8_t			sb_rgblklog;
+	uint8_t			sb_pad[7];
 };
 ----
 *sb_magicnum*::
@@ -413,6 +417,12 @@ to have a slightly higher level of redundancy over the shape of the inode
 btrees, and decreases the amount of time to compute the metadata B+tree
 preallocations at mount time.
 
+| +XFS_SB_FEAT_RO_COMPAT_RTSB+ |
+Realtime superblock.  The first rt extent in rt group zero contains a superblock
+header that can be used to identify the realtime device.  See the section about
+the xref:Realtime_Group_Superblocks[realtime group superblocks] for more
+information.
+
 |=====
 
 *sb_features_incompat*::
@@ -476,6 +486,10 @@ pointers] for more information.
 Metadata directory tree.  See the section about the xref:Metadata_Directories[
 metadata directory tree] for more information.
 
+| +XFS_SB_FEAT_INCOMPAT_RTGROUPS+ |
+Realtime allocation groups.  See the section about the xref:Realtime_Groups[
+realtime groups] for more information.
+
 |=====
 
 *sb_features_log_incompat*::
@@ -514,6 +528,23 @@ If the +XFS_SB_FEAT_RO_INCOMPAT_METADIR+ feature is set, this field points to
 the inode of the root directory of the metadata directory tree.
 This field is zero otherwise.
 
+*sb_rgcount*::
+Count of realtime groups in the filesystem, if the
++XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled.
+
+*sb_rgextents*::
+Maximum number of realtime extents that can be contained within a realtime
+group, if the +XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled.
+
+*sb_rgblklog*::
+If the +XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled, this is the log~2~
+value of +sb_rgextents+ * +sb_rextsize+ (rounded up). This value is used to
+generate absolute block numbers defined in extent maps from the segmented
++xfs_rtblock_t+ values.
+
+*sb_pad[7]*::
+Zeroes, if the +XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled.
+
 === xfs_db Superblock Example
 
 A filesystem is made on a single disk with the following command:
design/XFS_Filesystem_Structure/common_types.asciidoc +3 -1
index 51909be384e273..34cdfdaeccf848 100644
@@ -43,7 +43,9 @@ Unsigned 64 bit raw filesystem block number.
 
 *xfs_rtblock_t*::
 Unsigned 64 bit extent number in the xref:Real-time_Devices[real-time]
-sub-volume.
+sub-volume.  If the +XFS_SB_FEAT_INCOMPAT_METADIR+ feature is enabled, these
+values combine an xref:Realtime_Groups[rtgroup number] and block offset into
+the realtime group.
 
 *xfs_fileoff_t*::
 Unsigned 64 bit block offset into a file.
design/XFS_Filesystem_Structure/internal_inodes.asciidoc +2 -0
index 68c86d30ff8206..5f4d62201cbd67 100644
@@ -21,6 +21,8 @@ of those inodes have been deallocated and may be reused by future features.
 [options="header"]
 |=====
 | Metadata File                                  | Location
+| xref:Real-Time_Bitmap_Inode[Realtime Bitmap]   | /rtgroups/*.bitmap
+| xref:Real-Time_Summary_Inode[Realtime Summary] | /rtgroups/*.summary
 |=====
 
 Metadata files are flagged by the +XFS_DIFLAG2_METADATA+ flag in the
design/XFS_Filesystem_Structure/magic.asciidoc +3 -0
index 60952aeb876ff5..5da29b9ef9f3a8 100644
@@ -45,9 +45,12 @@ relevant chapters.  Magic numbers tend to have consistent locations:
 | +XFS_ATTR3_LEAF_MAGIC+	| 0x3bee	|     	| xref:Leaf_Attributes[Leaf Attribute], v5 only
 | +XFS_ATTR3_RMT_MAGIC+		| 0x5841524d	| XARM	| xref:Remote_Values[Remote Attribute Value], v5 only
 | +XFS_RMAP_CRC_MAGIC+		| 0x524d4233	| RMB3	| xref:Reverse_Mapping_Btree[Reverse Mapping B+tree], v5 only
+| +XFS_RTBITMAP_MAGIC+		| 0x424D505A	| BMPZ	| xref:Real-Time_Bitmap_Inode[Real-Time Bitmap], metadir only
+| +XFS_RTSUMMARY_MAGIC+		| 0x53554D59	| SUMY	| xref:Real-Time_Summary_Inode[Real-Time Summary], metadir only
 | +XFS_RTRMAP_CRC_MAGIC+	| 0x4d415052	| MAPR	| xref:Real_time_Reverse_Mapping_Btree[Real-Time Reverse Mapping B+tree], v5 only
 | +XFS_REFC_CRC_MAGIC+		| 0x52334643	| R3FC	| xref:Reference_Count_Btree[Reference Count B+tree], v5 only
 | +XFS_MD_MAGIC+		| 0x5846534d	| XFSM	| xref:Metadata_Dumps[Metadata Dumps]
+| +XFS_RTSB_MAGIC+		| 0x46726F67	| Frog	| xref:Realtime_Groups[Realtime Groups]
 |=====
 
 The magic numbers for log items are at offset zero in each log item, but items
design/XFS_Filesystem_Structure/ondisk_inode.asciidoc +2 -0
index 02ec0d12bb57e5..e28929907147b7 100644
@@ -199,6 +199,8 @@ directory tree.
 [source, c]
 ----
 enum xfs_metafile_type {
+     XFS_METAFILE_RTBITMAP,
+     XFS_METAFILE_RTSUMMARY,
 };
 ----
 
design/XFS_Filesystem_Structure/realtime.asciidoc +344 -0

354 lines not on this page · raw message

#14 lore raw

Re: [PATCH 2/4] design: document realtime groups

design/XFS_Filesystem_Structure/allocation_groups.asciidoc struct xfs_sb · line 112

Quoted 10 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +4 -0
@@ -106,6 +106,10 @@ struct xfs_sb
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
 	xfs_ino_t		sb_metadirino;
+	xfs_rgnumber_t		sb_rgcount;
+	xfs_rgblock_t		sb_rgextents;
+	uint8_t			sb_rgblklog;
+	uint8_t			sb_pad[7];

And following on with my ranting about existing bits theme from the previous review: why are we documenting the in-memory xfs_sb here and not the on-disk xfs_dsb?

design/XFS_Filesystem_Structure/allocation_groups.asciidoc to have a slightly higher level of redundancy over the shape of the inode · line 424

Quoted 5 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +5 -0
+| +XFS_SB_FEAT_RO_COMPAT_RTSB+ |
+Realtime superblock.  The first rt extent in rt group zero contains a superblock
+header that can be used to identify the realtime device.  See the section about
+the xref:Realtime_Group_Superblocks[realtime group superblocks] for more
+information.

This is actually gone now.

design/XFS_Filesystem_Structure/allocation_groups.asciidoc If the +XFS_SB_FEAT_RO_INCOMPAT_METADIR+ feature is set, this field points to · line 533

Quoted 3 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +3 -0
+*sb_rgcount*::
+Count of realtime groups in the filesystem, if the
++XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled.

... will be zero if XFS_SB_FEAT_INCOMPAT_RTGROUPS is set, but no realtime subvolume exists

?

#15 lore raw

Re: [PATCH 2/4] design: document realtime groups

Quoted 6 lines · Christoph Hellwig
Quoted 6 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +0 -0
@@ -106,6 +106,10 @@ struct xfs_sb
 	xfs_lsn_t		sb_lsn;
 	uuid_t			sb_meta_uuid;
 	xfs_ino_t		sb_metadirino;
Quoted 8 lines

design/XFS_Filesystem_Structure/allocation_groups.asciidoc struct xfs_sb · line 112

Quoted 4 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +4 -0
+	xfs_rgnumber_t		sb_rgcount;
+	xfs_rgblock_t		sb_rgextents;
+	uint8_t			sb_rgblklog;
+	uint8_t			sb_pad[7];

And following on with my ranting about existing bits theme from the previous review: why are we documenting the in-memory xfs_sb here and not the on-disk xfs_dsb?

<nod> will clean that one up too.

Quoted 7 lines

design/XFS_Filesystem_Structure/allocation_groups.asciidoc to have a slightly higher level of redundancy over the shape of the inode · line 424

Quoted 5 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +5 -0
+| +XFS_SB_FEAT_RO_COMPAT_RTSB+ |
+Realtime superblock.  The first rt extent in rt group zero contains a superblock
+header that can be used to identify the realtime device.  See the section about
+the xref:Realtime_Group_Superblocks[realtime group superblocks] for more
+information.

This is actually gone now.

Oops, will remove that one.

Quoted 8 lines

design/XFS_Filesystem_Structure/allocation_groups.asciidoc If the +XFS_SB_FEAT_RO_INCOMPAT_METADIR+ feature is set, this field points to · line 533

Quoted 3 lines
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +3 -0
+*sb_rgcount*::
+Count of realtime groups in the filesystem, if the
++XFS_SB_FEAT_INCOMPAT_RTGROUPS+ feature is enabled.

... will be zero if XFS_SB_FEAT_INCOMPAT_RTGROUPS is set, but no realtime subvolume exists

?

Yeah, that's a good thing to note. I'll also s/RTGROUPS/METADIR/ since the rtgroups feature bit is also gone yet I seem to have missed this one. :(

--D

#16 lore raw

PATCH 3/4 design: document metadata directory tree quota changes

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Document the changes to the ondisk quota metadata that came in with metadata directory trees.

---
 .../allocation_groups.asciidoc                     |    3 +++
 .../internal_inodes.asciidoc                       |    3 +++
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |    3 +++
 3 files changed, 9 insertions(+)
design/XFS_Filesystem_Structure/allocation_groups.asciidoc +3 -0
index 9f92be49a7a095..86daf9cdd30a0c 100644
@@ -293,6 +293,9 @@ Quota flags. It can be a combination of the following flags:
 | +XFS_PQUOTA_CHKD+		| Project quotas have been checked.
 |=====
 
+If the +XFS_SB_FEAT_INCOMPAT_METADIR+ feature is enabled, the +sb_qflags+ field
+will persist across mounts if no quota mount options are provided.
+
 *sb_flags*::
 Miscellaneous flags.
 
design/XFS_Filesystem_Structure/internal_inodes.asciidoc +3 -0
index 5f4d62201cbd67..40eb57233ce7c0 100644
@@ -21,6 +21,9 @@ of those inodes have been deallocated and may be reused by future features.
 [options="header"]
 |=====
 | Metadata File                                  | Location
+| xref:Quota_Inodes[User Quota]                  | /quota/user
+| xref:Quota_Inodes[Group Quota]                 | /quota/group
+| xref:Quota_Inodes[Project Quota]               | /quota/project
 | xref:Real-Time_Bitmap_Inode[Realtime Bitmap]   | /rtgroups/*.bitmap
 | xref:Real-Time_Summary_Inode[Realtime Summary] | /rtgroups/*.summary
 |=====
design/XFS_Filesystem_Structure/ondisk_inode.asciidoc +3 -0
index e28929907147b7..6e52e5fd3d6c1e 100644
@@ -199,6 +199,9 @@ directory tree.
 [source, c]
 ----
 enum xfs_metafile_type {
+     XFS_METAFILE_USRQUOTA,
+     XFS_METAFILE_GRPQUOTA,
+     XFS_METAFILE_PRJQUOTA,
      XFS_METAFILE_RTBITMAP,
      XFS_METAFILE_RTSUMMARY,
 };
#18 lore raw

PATCH 4/4 design: update metadump v2 format to reflect rt dumps

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Update the metadump v2 format documentation to add realtime device dumps.

---
 design/XFS_Filesystem_Structure/metadump.asciidoc |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
design/XFS_Filesystem_Structure/metadump.asciidoc +11 -1
index a32d6423ea6e75..226622c0d2f20e 100644
@@ -119,7 +119,16 @@ Dump contains external log contents.
 |=====
 
 *xmh_incompat_flags*::
-Must be zero.
+A combination of the following flags:
+
+.Metadump v2 incompat flags
+[options="header"]
+|=====
+| Flag				| Description
+| +XFS_MD2_INCOMPAT_RTDEVICE+	|
+Dump contains realtime device contents.
+
+|=====
 
 *xmh_reserved*::
 Must be zero.
@@ -143,6 +152,7 @@ Bits 55-56 determine the device from which the metadata dump data was extracted.
 | Value		| Description
 | 0		| Data device
 | 1		| External log
+| 2		| Realtime device
 |=====
 
 The lower 54 bits determine the device address from which the dump data was