Skip to contentSkip to conversation

xfs: Large extent counters

GIT PULL Chandan Babu R 2 participants 3 messages Apr 18, 2022 – Apr 22, 2022

lore
Share & tools
lore
#0 lore raw

Hi Dave,

The following changes since commit ce522ba9ef7e2d9fb22a39eb3371c0c64e2a433e:

Linux 5.18-rc2 (2022-04-10 14:21:36 -1000)

are available in the Git repository at:

https://github.com/chandanr/linux.git tags/large-extent-counters-v9

for you to fetch changes up to 973ac0eb3a7dfedecd385bd2b48b12e62a0492f2:

xfs: Add XFS_SB_FEAT_INCOMPAT_NREXT64 to the list of supported flags (2022-04-13 07:02:45 +0000)

----------------------------------------------------------------
xfs: Large extent counters

The commit xfs: fix inode fork extent count overflow (3f8a4f1d876d3e3e49e50b0396eaffcc4ba71b08) mentions that 10 billion data fork extents should be possible to create. However the corresponding on-disk field has a signed 32-bit type. Hence this patchset extends the per-inode data fork extent counter to 64 bits (out of which 48 bits are used to store the extent count).

Also, XFS has an attribute fork extent counter which is 16 bits wide. A workload that,
1. Creates 1 million 255-byte sized xattrs,
2. Deletes 50% of these xattrs in an alternating manner,
3. Tries to insert 400,000 new 255-byte sized xattrs causes the xattr extent counter to overflow.

Dave tells me that there are instances where a single file has more than 100 million hardlinks. With parent pointers being stored in xattrs, we will overflow the signed 16-bits wide attribute extent counter when large number of hardlinks are created. Hence this patchset extends the on-disk field to 32-bits.

The following changes are made to accomplish this,
1. A 64-bit inode field is carved out of existing di_pad and di_flushiter fields to hold the 64-bit data fork extent counter.
2. The existing 32-bit inode data fork extent counter will be used to hold the attribute fork extent counter.
3. A new incompat superblock flag to prevent older kernels from mounting the filesystem.

The patchset has been tested by executing fstests with the following mkfs.xfs options,
1. -m crc=0 -b size=1k
2. -m crc=0 -b size=4k
3. -m crc=0 -b size=512
4. -m rmapbt=1,reflink=1 -b size=1k
5. -m rmapbt=1,reflink=1 -b size=4k

Each of the above test scenarios was executed on the following combinations (For V4 FS test scenario, the last combination was omitted).

------------------------------------
Xfsprogs Kernel
------------------------------------

 Unpatched                  Patched
 Patched (disable nrext64)  Patched
 Patched (enable nrext64)   Patched

------------------------------------

I have also written tests to check if the correct extent counter fields are updated with/without the new incompat flag and to verify upgrading older fs instances to support large extent counters. I have also fixed xfs/270 test to work with the new code base.

----------------------------------------------------------------
Chandan Babu R (19):
xfs: Move extent count limits to xfs_format.h
xfs: Define max extent length based on on-disk format definition
xfs: Introduce xfs_iext_max_nextents() helper
xfs: Use xfs_extnum_t instead of basic data types
xfs: Introduce xfs_dfork_nextents() helper
xfs: Use basic types to define xfs_log_dinode's di_nextents and di_anextents
xfs: Promote xfs_extnum_t and xfs_aextnum_t to 64 and 32-bits respectively
xfs: Introduce XFS_SB_FEAT_INCOMPAT_NREXT64 and associated per-fs feature bit
xfs: Introduce XFS_FSOP_GEOM_FLAGS_NREXT64
xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers
xfs: Use uint64_t to count maximum blocks that can be used by BMBT
xfs: Introduce macros to represent new maximum extent counts for data/attr forks
xfs: Replace numbered inode recovery error messages with descriptive ones
xfs: Introduce per-inode 64-bit extent counters
xfs: Directory's data fork extent counter can never overflow
xfs: Conditionally upgrade existing inodes to use large extent counters
xfs: Decouple XFS_IBULK flags from XFS_IWALK flags
xfs: Enable bulkstat ioctl to support 64-bit per-inode extent counters
xfs: Add XFS_SB_FEAT_INCOMPAT_NREXT64 to the list of supported flags

 fs/xfs/libxfs/xfs_alloc.c       |   2 +-
 fs/xfs/libxfs/xfs_attr.c        |   3 +
 fs/xfs/libxfs/xfs_bmap.c        | 109 +++++++++++++------------------
 fs/xfs/libxfs/xfs_bmap_btree.c  |   9 ++-
 fs/xfs/libxfs/xfs_da_btree.h    |   1 +
 fs/xfs/libxfs/xfs_da_format.h   |   1 +
 fs/xfs/libxfs/xfs_dir2.c        |   8 +++
 fs/xfs/libxfs/xfs_format.h      | 104 ++++++++++++++++++++++++++---
 fs/xfs/libxfs/xfs_fs.h          |  25 +++++--
 fs/xfs/libxfs/xfs_ialloc.c      |   2 +
 fs/xfs/libxfs/xfs_inode_buf.c   |  83 ++++++++++++++++++-----
 fs/xfs/libxfs/xfs_inode_fork.c  |  39 +++++++++--
 fs/xfs/libxfs/xfs_inode_fork.h  |  76 ++++++++++++++++++----
 fs/xfs/libxfs/xfs_log_format.h  |  33 ++++++++--
 fs/xfs/libxfs/xfs_sb.c          |   5 ++
 fs/xfs/libxfs/xfs_trans_resv.c  |  11 ++--
 fs/xfs/libxfs/xfs_types.h       |  11 +---
 fs/xfs/scrub/bmap.c             |   2 +-
 fs/xfs/scrub/inode.c            |  20 +++---
 fs/xfs/xfs_bmap_item.c          |   2 +
 fs/xfs/xfs_bmap_util.c          |  27 ++++++--
 fs/xfs/xfs_dquot.c              |   3 +
 fs/xfs/xfs_inode.c              |  59 ++---------------
 fs/xfs/xfs_inode.h              |   5 ++
 fs/xfs/xfs_inode_item.c         |  23 +++++--
 fs/xfs/xfs_inode_item_recover.c | 141 ++++++++++++++++++++++++++++------------
 fs/xfs/xfs_ioctl.c              |   3 +
 fs/xfs/xfs_iomap.c              |  33 ++++++----
 fs/xfs/xfs_itable.c             |  15 ++++-
 fs/xfs/xfs_itable.h             |   5 +-
 fs/xfs/xfs_iwalk.h              |   2 +-
 fs/xfs/xfs_mount.h              |   2 +
 fs/xfs/xfs_reflink.c            |   5 ++
 fs/xfs/xfs_rtalloc.c            |   3 +
 fs/xfs/xfs_super.c              |   4 ++
 fs/xfs/xfs_symlink.c            |   5 --
 fs/xfs/xfs_trace.h              |   4 +-
 37 files changed, 607 insertions(+), 278 deletions(-)
Signature · 1 line
chandan
#1 lore raw

Hi Chandan,

Quoted 9 lines · Chandan Babu R

Hi Dave,

The following changes since commit ce522ba9ef7e2d9fb22a39eb3371c0c64e2a433e:

Linux 5.18-rc2 (2022-04-10 14:21:36 -1000)

are available in the Git repository at:

https://github.com/chandanr/linux.git tags/large-extent-counters-v9

I've pulled this down and it seems fine, but there's problem I can't solve directly: the tag isn't signed so I can't tell if the tree I've pulled is unmolested.

If you were asking me to pull from git.kernel.org, that wouldn't be a problem. But I don't trust github to be safe - forget about hackers, github's owners have demonstrated they will modify repositories directly if it is convenient for them to do so. i.e. they have overwritten hosted trees without the author's and/or copyright owner's consent and locked out the owner(s) of the code from ever being able to update their tree again.

As such, I don't consider github to be a trustworthy source, and so if you are going to use it for pull requests I need the tag to be signed with you gpg key so that I can verify the commits I'm pulling match the ones you pushed to the hosting site.

Can you post on #xfs or DM on oftc the last five commits (--one-line format is fine) from your local branch that you pushed to github so that I have an OOB confirmation that the commit IDs I've pulled match your original commits?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 30 lines · Dave Chinner

Hi Chandan,

Quoted 9 lines · Chandan Babu R

Hi Dave,

The following changes since commit ce522ba9ef7e2d9fb22a39eb3371c0c64e2a433e:

Linux 5.18-rc2 (2022-04-10 14:21:36 -1000)

are available in the Git repository at:

https://github.com/chandanr/linux.git tags/large-extent-counters-v9

I've pulled this down and it seems fine, but there's problem I can't solve directly: the tag isn't signed so I can't tell if the tree I've pulled is unmolested.

If you were asking me to pull from git.kernel.org, that wouldn't be a problem. But I don't trust github to be safe - forget about hackers, github's owners have demonstrated they will modify repositories directly if it is convenient for them to do so. i.e. they have overwritten hosted trees without the author's and/or copyright owner's consent and locked out the owner(s) of the code from ever being able to update their tree again.

As such, I don't consider github to be a trustworthy source, and so if you are going to use it for pull requests I need the tag to be signed with you gpg key so that I can verify the commits I'm pulling match the ones you pushed to the hosting site.

Quoted 4 lines

Can you post on #xfs or DM on oftc the last five commits (--one-line format is fine) from your local branch that you pushed to github so that I have an OOB confirmation that the commit IDs I've pulled match your original commits?

Hi Dave,

I have sent the details of the last five commits via DM on OFTC. Please let me know if you haven't received it.

Signature · 1 line
chandan