Skip to content

NFS Direct: align misaligned DIO for LOCALIO

Mike Snitzer linux-nfs 8 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v11 Latest7 patches 3 repliesIn review
  2. v107 patches 14 repliesSuperseded
  3. v97 patches 9 repliesSuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v89 patches0 repliesSuperseded
v711 patches0 repliesSuperseded
v67 patches0 repliesSuperseded
v27 patches6 repliesSuperseded
v1 RFC6 patches2 repliesSuperseded

Changes in v11 from the cover letter

- Fixed patch 6's nfstrace.h compiler errors due to #include order in nfs2xdr.c
- Rebased patches 2 and 5 to reduce churn/indentation for EINVAL error logging.
- Cc Chuck and Jeff for review of patch 2's fs/nfsd/localio.c change to add .nfsd_file_dio_alignment callback to nfsd_localio_operations

Patches in v11 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7nfs/localio: make trace_nfs_local_open_fh more usefulfs/nfs/localio.c fs/nfs/nfstrace.h+6 -50 replies
2/7nfs/localio: avoid issuing misaligned IO using O_DIRECTfs/nfs/localio.c fs/nfsd/localio.c include/linux/nfslocalio.h+68 -103 replies
2/7nfs/localio: avoid issuing misaligned IO using O_DIRECTfs/nfs/localio.c fs/nfsd/localio.c include/linux/nfslocalio.h+70 -70 replies
3/7nfs/localio: refactor iocb and iov_iter_bvec initializationfs/nfs/localio.c+33 -370 replies
4/7nfs/localio: refactor iocb initializationfs/nfs/localio.c+56 -390 replies
5/7nfs/localio: add proper O_DIRECT support for READ and WRITEfs/nfs/localio.c+201 -460 replies
6/7nfs/localio: add tracepoints for misaligned DIO READ and WRITE supportfs/nfs/internal.h fs/nfs/localio.c fs/nfs/nfs2xdr.c +2 more+90 -130 replies
7/7NFS: add basic STATX_DIOALIGN and STATX_DIO_READ_ALIGN supportfs/nfs/inode.c+15 -00 replies

Reviews trailers per patch

2 of 8 patches have a Reviewed-by or Acked-by.

PatchJeff Layton
1/7
2/7R-b
2/7R-b
3/7
4/7
5/7
6/7
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v11 thread

Cover letter [PATCH v11 0/7] NFS Direct: align misaligned DIO for LOCALIO

Hi Anna,

NFS and LOCALIO in particular benefit from avoiding the page cache for workloads that have a working set that is significantly larger than available system memory. NFS DIRECT makes it possible to always enable LOCALIO to use O_DIRECT even if the IO is not DIO-aligned.

This patchset's changes are focused on NFS LOCALIO (fs/nfs/localio.c); as such they will not impact NFS at all unless CONFIG_NFS_LOCALIO=y

These changes have been tested quite a lot, using workloads from load generators fio and dt. But also using MLperf benchmark suite's npz loader library (written in python) that creates a particularly nasty 1MB IO pattern which is DIO-aligned on disk but _not_ in memory (so it triggers the need to fallback from DIO to buffered IO). I'm happy to share more specifics on test tools and commandlines if needed.

Changes since v10 (09.17.2025):
- Fixed patch 6's nfstrace.h compiler errors due to #include order in nfs2xdr.c
- Rebased patches 2 and 5 to reduce churn/indentation for EINVAL error logging.
- Cc Chuck and Jeff for review of patch 2's fs/nfsd/localio.c change to add .nfsd_file_dio_alignment callback to nfsd_localio_operations

Changes since v9 (09.15.2025):
- Updated patch 3 to make bvec init while loop similar to nfsd_iter_read()
- To support both sync and async completion of DIO (sync completion will happen if underlying driver is ramdisk, etc): push DIO's nfs_local_pgio_done() down from nfs_local_{read,write}_done to nfs_local_{read,write}_aio_complete

Changes since v8 (08.15.2025)
- Removed all fs/nfs/direct.c changes and pushed this misaligned DIO handling down to LOCALIO where it belongs.
- Updated all commit headers to reflect changes to code.
- Because misaligned DIO is now handled properly in LOCALIO, removed the nfs modparam 'localio_O_DIRECT_semantics' that was added during v6.14 to require users opt-in to the requirement that all O_DIRECT be properly DIO-aligned.
- Enhanced LOCALIO's DIO WRITE code to handle potential for VFS to return -ENOTBLK if it fails to invalidate page cache on WRITE.
- Verified various test IO workloads function as expected; including a misaligned DIO test that failed with the previous v8 direct.c implementation.
- Verified performance remains high with LOCALIO on fast NVMe.
- Verified sparse and checkpatch.pl are clean.

Earlier changelog was provided in v8's 0th patch header, see:
https://lore.kernel.org/linux-nfs/[email protected]/

All review appreciated, thanks.
Mike

Mike Snitzer (7):
nfs/localio: make trace_nfs_local_open_fh more useful nfs/localio: avoid issuing misaligned IO using O_DIRECT nfs/localio: refactor iocb and iov_iter_bvec initialization nfs/localio: refactor iocb initialization
nfs/localio: add proper O_DIRECT support for READ and WRITE nfs/localio: add tracepoints for misaligned DIO READ and WRITE support
NFS: add basic STATX_DIOALIGN and STATX_DIO_READ_ALIGN support

 fs/nfs/inode.c             |  15 ++
 fs/nfs/internal.h          |  10 +
 fs/nfs/localio.c           | 405 ++++++++++++++++++++++++++++---------
 fs/nfs/nfs2xdr.c           |   2 +-
 fs/nfs/nfs3xdr.c           |   2 +-
 fs/nfs/nfstrace.h          |  76 ++++++-
 fs/nfsd/localio.c          |  11 +
 include/linux/nfslocalio.h |   2 +
 8 files changed, 421 insertions(+), 102 deletions(-)
Signature · 1 line
2.44.0