RFC fs: range-based admission for generation-drained DIO

Hi all,

This patch series attempts to resolve a major source of contention between concurrent readers and writers on the same inode, specifically when those writers are punching holes in the file or gapped appends provided that readers aren't overlapping these operations. There's some interface changes I needed just for this initial exclusive IOLOCK removal because DIO completion needs to cooperate and that lives in generic layers.

I believe the patch set to be complete and well-tested. The details performance numbers are available in my earlier XFS post:
https://lore.kernel.org/linux-xfs/[email protected]/T/#t
This patch series is identical, just widening the group.

Performance measurements on XFS available in the previous posting indicate substantial read throughput + P99 latency improvements under concurrent read/write contention with no negative impact to pure read scenarios. Fixing all sources of contention even manages to improve pure read workloads by ~2-5%.

Technical overview
------------------

To opt-in, the filesystem serializes compatible punch/gapped operations with a per-inode drain mutex (only for filesystems that opt-in). It publishes an exclusion range through a seqcount latch and flips the DIO generation counter, capturing the outstanding request count. Completion of old-generation requests retires that finite cohort to wake the drainer; this is similar semantics to today with the exception that reads can continue during this exclusive drain mutex hold provided they don't overlap the range (the writer doesn't wait for them to drain either since it's guaranteed they don't overlap). Overlapping reads queue up behind the drain mutex.

Filesystems not using the new interface don't pay a cost.

Not all cases are eligible for this fast-path and those degrade to the old exclusive IOLOCK + drain machinery, including DAX, realtime and unaligned punches; the goal is to remain conservative for now while correctly solving scaling bottlenecks for the common cases. I've tried to make sure there's good additional test coverage for the new code paths.

I expect the same concept would also benefit ext4 and Btrfs which similarly hold exclusive IOLOCKs; I have only double-checked that they broadly follow the similar pattern but I'm not 100% sure if the interface introduced here strictly applies the same to solve the same scaling problem. I could imagine small adjustments being necessary. Like XFS they have additional sources of contention present beyond what this specific patch series solves and like XFS I expect filesystem-specific optimizations will need to be written; it's possible there could be a shared mechanism to encode some of this but I think all contention sources for at least two or three filesystems need to be solved before generic interfaces are codified.

In particular, I would appreciate feedback on:

* Whether the generic inode DIO accounting and iomap interfaces, with filesystem-private lifetime management through ->dio_drain_complete is correct.
* The generation/count encoding and ordering guarantees.
* The XFS lock/revalidation contract and whether my analysis is correct that ext4/Btrfs would benefit from the same interface or if different hooks are needed.

The base is 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4. The patch ordering is generic accounting, iomap ownership, KUnit coverage, XFS admission, aligned hole punching, then post-EOF zeroing.

Thanks,
Vitali

Vitali Lovich (6):
fs: implement range-based generation-coordinated DIO draining
iomap: support generation accounting and caller-registered DIO
fs: add KUnit coverage for generation-based DIO draining
xfs: add range-drain state and latch-based DIO admission
xfs: use range-based DIO draining for aligned hole punches
xfs: use range-based DIO draining for post-EOF zeroing

 Documentation/filesystems/locking.rst         |  89 +-
 Documentation/filesystems/vfs.rst             |  41 +
 fs/Kconfig                                    |  13 +
 fs/Makefile                                   |   1 +
 fs/ext4/file.c                                |   2 +-
 fs/inode.c                                    | 120 ++-
 fs/iomap/direct-io.c                          |  56 +-
 fs/tests/dio_drain_kunit.c                    | 893 ++++++++++++++++++
 fs/xfs/Kconfig                                |   9 +
 fs/xfs/Makefile                               |   2 +
 fs/xfs/tests/xfs_dio_test.c                   | 190 ++++
 fs/xfs/xfs_bmap_util.c                        |   9 +-
 fs/xfs/xfs_dio_test.h                         |  32 +
 fs/xfs/xfs_file.c                             | 357 ++++++-
 fs/xfs/xfs_icache.c                           |   2 +
 fs/xfs/xfs_inode.c                            |  10 +
 fs/xfs/xfs_inode.h                            |   1 +
 fs/xfs/xfs_iops.c                             |   9 +
 fs/xfs/xfs_reflink.c                          |   2 +
 fs/xfs/xfs_reflink.h                          |   2 +-
 fs/xfs/xfs_super.c                            |   2 +
 include/linux/fs.h                            |  97 +-
 include/linux/iomap.h                         |  60 +-
 .../litmus-tests/DIO-register-first.litmus    |  36 +
 tools/testing/selftests/filesystems/Makefile  |   6 +
 .../selftests/filesystems/xfs_dio_admission.c | 380 ++++++++
 .../selftests/filesystems/xfs_dio_drain.c     | 392 ++++++++
 .../selftests/filesystems/xfs_dio_remap.c     | 391 ++++++++
 28 files changed, 3116 insertions(+), 88 deletions(-)
 create mode 100644 fs/tests/dio_drain_kunit.c
 create mode 100644 fs/xfs/tests/xfs_dio_test.c
 create mode 100644 fs/xfs/xfs_dio_test.h
 create mode 100644 tools/memory-model/litmus-tests/DIO-register-first.litmus
 create mode 100644 tools/testing/selftests/filesystems/xfs_dio_admission.c
 create mode 100644 tools/testing/selftests/filesystems/xfs_dio_drain.c
 create mode 100644 tools/testing/selftests/filesystems/xfs_dio_remap.c

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

2.56.0