Skip to contentSkip to conversation

NFSD: keep direct-mode I/O out of the page cache

PATCH Superseded Mike Snitzer 1 participant 11 messages Oct 1

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover NFSD: keep direct-mode I/O out of the page cache

Hi,

This series changes NFSD's direct write path: io_cache_write modes where an aligned WRITE goes to the filesystem as O_DIRECT, with DONTCACHE for the parts a misaligned WRITE cannot issue directly.

1/9 marks the direct middle of a split WRITE IOCB_DONTCACHE, for when XFS re-issues it buffered (-ENOTBLK). It carries Fixes: and rewrites the stale nfsd-io-modes.rst sentence in place, so a stable tree carrying only this patch reads consistently.

2/9 splits a misaligned WRITE only for a direct middle of at least direct_misaligned_num_pages (default 2). 3/9 (Jonathan) adds direct_misaligned_dontcache, which selects whether the parts that are not O_DIRECT are DONTCACHE or cached.

4/9 stops expanding a READ smaller than its alignment into a full aligned device read.

5/9 persists a synchronous direct-mode WRITE once, after all of its segments, instead of once per segment. 6/9 keeps the page two misaligned WRITEs share until both have written it: 704 device reads for 42895 WRITEs where there were 45144 for 45664. 7/9 adds tracepoints for how each direct-mode READ and WRITE was serviced.

8/9 and 9/9 are separable and come last for that reason. 8/9 is Chuck's "Enable return of an updated stable_how to NFS clients", reworked onto @iocb_flags. 9/9 adds two opt-in io_cache_write modes that persist every direct-mode WRITE before replying and report NFS_DATA_SYNC or NFS_FILE_SYNC, so the client sends no COMMIT. A client can get the same result by sending FILE_SYNC itself, so 1-7 stand without them.

The series applies to cel/nfsd-testing (32eb1a60b456) and each patch builds with W=1 with no warnings.

Changes since v2:
- Rebased onto cel/nfsd-testing; 8/9 now applies to the xdrgen nfsd_proc_write().
- 1/9: replaces the "DONTCACHE buffered IO is _not_ used" sentence instead of adding a paragraph that contradicted it.
- 2/9: the description now opens with the cost a small middle adds, says the IOCB_* flags only move (the write loop already set IOCB_DONTCACHE), and says which parts of Christoph's diff are taken and why the DONTCACHE policy and the memory-misaligned collapse are not. The DONTCACHE explanation is one comment at no_dio; the debugfs comment sits on the debugfs_create_u32() call; "three segments" is now "two or three".
- Reordered: direct_misaligned_dontcache moved up to 3/9, beside direct_misaligned_num_pages, and the boundary-page patch (now 6/9) adds its own fields. The two stable_how patches moved to the end (8/9, 9/9), since nothing before them depends on them.
- 8/9: the async export option clears a local copy, so the reply to a FILE_SYNC or DATA_SYNC WRITE to an async export is never lowered (RFC 1813 3.3.7). The description says where NFSv3 and NFSv4 map the result, and no longer says O_DIRECT persists data.
- 9/9: retitled. The description says an UNSTABLE O_DIRECT write is not durable, that NFSD_IO_DIRECT is unchanged, and that the new modes cost a flush per WRITE. They are worth it only where each COMMIT covers about one WRITE.
- Comments and documentation trimmed throughout.

v2: https://lore.kernel.org/linux-nfs/[email protected]/
v1: https://lore.kernel.org/linux-nfs/[email protected]/

Thanks,
Mike

Chuck Lever (1):
NFSD: Enable return of an updated stable_how to NFS clients

Jonathan Flynn (1):
NFSD: add direct_misaligned_dontcache debugfs knob

Mike Snitzer (7):
NFSD: mark the direct middle of a split WRITE IOCB_DONTCACHE as well
NFSD: only split a direct-mode WRITE for a worthwhile direct middle
NFSD: do not use direct I/O for a READ smaller than its alignment
NFSD: persist a synchronous direct-mode WRITE once, after all of its segments
NFSD: keep boundary page of a split direct-mode WRITE until both writers complete
NFSD: add tracing for how direct-mode READ and WRITE are serviced
NFSD: add direct-mode WRITE settings that persist each WRITE

 .../filesystems/nfs/nfsd-io-modes.rst         |  97 ++++++-
 fs/nfsd/debugfs.c                             |  28 ++
 fs/nfsd/nfs3proc.c                            |  16 +-
 fs/nfsd/nfs4proc.c                            |  15 +-
 fs/nfsd/nfsd.h                                |   4 +
 fs/nfsd/nfsproc.c                             |   3 +-
 fs/nfsd/trace.h                               |  85 +++++++
 fs/nfsd/vfs.c                                 | 239 +++++++++++++++---
 fs/nfsd/vfs.h                                 |  18 +-
 fs/nfsd/xdr3.h                                |   2 +-
 10 files changed, 449 insertions(+), 58 deletions(-)

base-commit: 32eb1a60b456980761cf7a9cee8f907fdc08afb8

Signature · 1 line
2.52.0
#1 lore raw

PATCH v3 1/9 NFSD: mark the direct middle of a split WRITE IOCB_DONTCACHE as well

nfsd_write_dio_iters_init() issues the DIO-aligned middle of a misaligned WRITE with IOCB_DIRECT. A file system may service that request with buffered I/O instead: xfs_file_write_iter() retries through xfs_file_buffered_write() with the same kiocb when xfs_file_dio_write() returns -ENOTBLK, which iomap_dio_rw() does when it cannot invalidate page cache overlapping the range.

In a streaming misaligned WRITE workload with more than one request in flight this happens routinely. Request N's direct middle ends in the page where request N+1's buffered prefix starts. If N+1 dirties that page between N's writeback and N's invalidation, the invalidation fails and XFS re-issues all of N's middle, tens of kilobytes, as a cached buffered write that nothing ever drops.

Set IOCB_DONTCACHE on the middle segment alongside IOCB_DIRECT when the file system supports FOP_DONTCACHE. The direct path ignores the flag; a buffered fallback now drops its pages once written back.

nfsd-io-modes.rst said that DONTCACHE is not used for the misaligned segments, which the write loop already contradicted: every segment without IOCB_DIRECT gets IOCB_DONTCACHE when the file system supports it. Replace that sentence with one that covers all three segments.

Fixes 06c5c97293e3 ("NFSD: Implement NFSD_IO_DIRECT for NFS WRITE")
Assisted-by Claude:claude-fable-5-1
---
 Documentation/filesystems/nfs/nfsd-io-modes.rst | 8 +++++---
 fs/nfsd/vfs.c                                   | 3 +++
 2 files changed, 8 insertions(+), 3 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +5 -3
index 0fd6e82478fe6..60b0af9b7e49f 100644
@@ -126,9 +126,11 @@ Misaligned WRITE:
     middle and end as needed. The large middle segment is DIO-aligned
     and the start and/or end are misaligned. Buffered IO is used for the
     misaligned segments and O_DIRECT is used for the middle DIO-aligned
-    segment. DONTCACHE buffered IO is _not_ used for the misaligned
-    segments because using normal buffered IO offers significant RMW
-    performance benefit when handling streaming misaligned WRITEs.
+    segment. If the filesystem supports FOP_DONTCACHE, every segment is
+    marked DONTCACHE. The flag has no effect on the O_DIRECT segment
+    unless the filesystem services it with buffered IO instead, as XFS
+    does when it cannot invalidate page cache that overlaps the segment.
+    The iomap_dio_invalidate_fail trace event reports such a fallback.
 
 Tracing:
     The nfsd_read_direct trace event shows how NFSD expands any
fs/nfsd/vfs.c +3 -0
index 4584d5b94feed..1ccdac4693745 100644
@@ -1341,6 +1341,9 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	if (iov_iter_bvec_offset(&segments[nsegs].iter) & (mem_align - 1))
 		goto no_dio;
 	segments[nsegs].flags |= IOCB_DIRECT;
+	/* In case the file system falls back to buffered I/O (-ENOTBLK). */
+	if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE)
+		segments[nsegs].flags |= IOCB_DONTCACHE;
 	nsegs++;
 
 	if (suffix)

base-commit: 32eb1a60b456980761cf7a9cee8f907fdc08afb8

Signature · 1 line
2.52.0
#2 lore raw

PATCH v3 2/9 NFSD: only split a direct-mode WRITE for a worthwhile direct middle

In reply to Mike Snitzer

Splitting a misaligned direct-mode WRITE issues it as two or three writes instead of one, and the direct write first writes back and invalidates the page cache over its range. When the DIO-aligned middle is only a page or so, the WRITE is mostly its buffered prefix and suffix anyway, and the direct middle adds that overhead plus the invalidation that contends with a neighbouring WRITE's buffered segment in the page they share.

Add direct_misaligned_num_pages (debugfs, default 2) and do not split a WRITE that has a misaligned start or end and a middle smaller than that many pages. Such a WRITE is issued as a single buffered segment, like the other WRITEs NFSD does not split.

Also decide each segment's IOCB_* flags in nfsd_write_dio_iters_init() instead of in the write loop of nfsd_direct_write(). The flags do not
change: the loop already set IOCB_DONTCACHE on every segment without IOCB_DIRECT when the file system supports FOP_DONTCACHE.

The threshold and the move of the flag decisions come from Christoph's suggested diff [1]. Two of its choices are not taken. It issued the prefix, the suffix and a WRITE smaller than its alignment as cached buffered I/O, to leave partial pages in place for read-modify-write; this patch keeps the DONTCACHE policy already in the tree, and the next patch makes that policy selectable. It also split a WRITE whose payload is misaligned in memory and issued the middle as DONTCACHE buffered I/O; no part of such a WRITE can be direct I/O, so it is still issued whole and treated like every other WRITE that cannot be.

Document in nfsd-io-modes.rst when NFSD does not split a WRITE.

---
 .../filesystems/nfs/nfsd-io-modes.rst         | 13 ++++++
 fs/nfsd/debugfs.c                             | 11 +++++
 fs/nfsd/nfsd.h                                |  1 +
 fs/nfsd/vfs.c                                 | 41 +++++++++++--------
 4 files changed, 48 insertions(+), 18 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +13 -0
index 60b0af9b7e49f..0a67ce9244d38 100644
@@ -132,6 +132,19 @@ Misaligned WRITE:
     does when it cannot invalidate page cache that overlaps the segment.
     The iomap_dio_invalidate_fail trace event reports such a fallback.
 
+    If NFSD does not split a misaligned WRITE, it issues the whole WRITE
+    as a single DONTCACHE buffered IO (normal buffered IO if the
+    filesystem lacks FOP_DONTCACHE). NFSD does not split a WRITE when:
+
+    - the filesystem advertises no DIO alignment;
+    - the WRITE is smaller than the larger of the offset and memory
+      alignments;
+    - the WRITE has a misaligned start or end and its DIO-aligned middle
+      is smaller than /sys/kernel/debug/nfsd/direct_misaligned_num_pages
+      pages (default 2);
+    - the WRITE payload is not aligned in memory to the block device's
+      dma_alignment, so the middle cannot be O_DIRECT either.
+
 Tracing:
     The nfsd_read_direct trace event shows how NFSD expands any
     misaligned READ to the next DIO-aligned block (on either end of the
fs/nfsd/debugfs.c +11 -0
index 386fd1c54f527..0b3ddf28d2849 100644
@@ -140,6 +140,17 @@ void nfsd_debugfs_init(void)
 
 	debugfs_create_file("io_cache_write", 0644, nfsd_top_dir, NULL,
 			    &nfsd_io_cache_write_fops);
+
+	/*
+	 * /sys/kernel/debug/nfsd/direct_misaligned_num_pages
+	 *
+	 * Minimum size, in pages, of the DIO-aligned middle segment for NFSD
+	 * to split a misaligned direct-mode WRITE.  A WRITE with a misaligned
+	 * start or end and a smaller middle is issued as a single buffered
+	 * segment.  The default value of this setting is 2.
+	 */
+	debugfs_create_u32("direct_misaligned_num_pages", 0644, nfsd_top_dir,
+			   &nfsd_direct_misaligned_num_pages);
 #ifdef CONFIG_NFSD_V4
 	debugfs_create_bool("delegated_timestamps", 0644, nfsd_top_dir,
 			    &nfsd_delegts_enabled);
fs/nfsd/nfsd.h +1 -0
index 8e8763ae7035c..70219d26b7404 100644
@@ -145,6 +145,7 @@ enum {
 
 extern u64 nfsd_io_cache_read __read_mostly;
 extern u64 nfsd_io_cache_write __read_mostly;
+extern u32 nfsd_direct_misaligned_num_pages __read_mostly;
 
 bool nfsd_v4client(struct svc_rqst *rqstp);
 
fs/nfsd/vfs.c +23 -18

87 lines not on this page · raw message

Signature · 1 line
2.52.0
#3 lore raw

PATCH v3 3/9 NFSD: add direct_misaligned_dontcache debugfs knob

In reply to Mike Snitzer

From: Jonathan Flynn <[email protected]>

The parts of a direct-mode WRITE that are not O_DIRECT, the misaligned start and end of a split WRITE and a WRITE that is not split, are issued IOCB_DONTCACHE when the file system supports it. That suits the workloads a direct mode is chosen for, but it is a policy, not a
requirement: a workload that reads back or rewrites what it just wrote is better served by those pages staying cached.

Add /sys/kernel/debug/nfsd/direct_misaligned_dontcache (default Y). Set to N, those parts are issued as normal buffered I/O. The O_DIRECT middle keeps IOCB_DONTCACHE either way. It sits beside direct_misaligned_num_pages, which decides how much of a misaligned WRITE is O_DIRECT; this knob decides how the rest is cached.

[snitzer: switched from a module parameter to a debugfs knob, moved next to direct_misaligned_num_pages, documented in nfsd-io-modes.rst]

Assisted-by Claude:claude-opus-5[1m]
---
 .../filesystems/nfs/nfsd-io-modes.rst          | 18 +++++++++++++-----
 fs/nfsd/debugfs.c                              | 12 ++++++++++++
 fs/nfsd/nfsd.h                                 |  1 +
 fs/nfsd/vfs.c                                  | 13 +++++++++----
 4 files changed, 35 insertions(+), 9 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +13 -5
index 0a67ce9244d38..9b1a9e7b09cef 100644
@@ -126,11 +126,13 @@ Misaligned WRITE:
     middle and end as needed. The large middle segment is DIO-aligned
     and the start and/or end are misaligned. Buffered IO is used for the
     misaligned segments and O_DIRECT is used for the middle DIO-aligned
-    segment. If the filesystem supports FOP_DONTCACHE, every segment is
-    marked DONTCACHE. The flag has no effect on the O_DIRECT segment
-    unless the filesystem services it with buffered IO instead, as XFS
-    does when it cannot invalidate page cache that overlaps the segment.
-    The iomap_dio_invalidate_fail trace event reports such a fallback.
+    segment. If the filesystem supports FOP_DONTCACHE, the O_DIRECT
+    segment is marked DONTCACHE, and so are the misaligned segments
+    unless direct_misaligned_dontcache is N (below). The flag has no
+    effect on the O_DIRECT segment unless the filesystem services it
+    with buffered IO instead, as XFS does when it cannot invalidate page
+    cache that overlaps the segment. The iomap_dio_invalidate_fail trace
+    event reports such a fallback.
 
     If NFSD does not split a misaligned WRITE, it issues the whole WRITE
     as a single DONTCACHE buffered IO (normal buffered IO if the
@@ -145,6 +147,12 @@ Misaligned WRITE:
     - the WRITE payload is not aligned in memory to the block device's
       dma_alignment, so the middle cannot be O_DIRECT either.
 
+    Writing N to /sys/kernel/debug/nfsd/direct_misaligned_dontcache
+    (default Y) issues the start and end segments, and a WRITE that is
+    not split, as normal buffered IO instead of DONTCACHE, which suits a
+    workload that reads back or rewrites what it just wrote. The O_DIRECT
+    middle segment keeps its DONTCACHE flag either way.
+
 Tracing:
     The nfsd_read_direct trace event shows how NFSD expands any
     misaligned READ to the next DIO-aligned block (on either end of the
fs/nfsd/debugfs.c +12 -0
index 0b3ddf28d2849..d5b714dd0fbc6 100644
@@ -151,6 +151,18 @@ void nfsd_debugfs_init(void)
 	 */
 	debugfs_create_u32("direct_misaligned_num_pages", 0644, nfsd_top_dir,
 			   &nfsd_direct_misaligned_num_pages);
+
+	/*
+	 * /sys/kernel/debug/nfsd/direct_misaligned_dontcache
+	 *
+	 *   Y: the parts of a direct-mode WRITE that are not O_DIRECT use
+	 *      DONTCACHE buffered IO when the file system supports it
+	 *   N: those parts use normal buffered IO
+	 *
+	 * The default value of this setting is Y.
+	 */
+	debugfs_create_bool("direct_misaligned_dontcache", 0644, nfsd_top_dir,
+			    &nfsd_direct_misaligned_dontcache);
 #ifdef CONFIG_NFSD_V4
 	debugfs_create_bool("delegated_timestamps", 0644, nfsd_top_dir,
 			    &nfsd_delegts_enabled);
fs/nfsd/nfsd.h +1 -0
index 70219d26b7404..1c17f96012523 100644
@@ -146,6 +146,7 @@ enum {
 extern u64 nfsd_io_cache_read __read_mostly;
 extern u64 nfsd_io_cache_write __read_mostly;
 extern u32 nfsd_direct_misaligned_num_pages __read_mostly;
+extern bool nfsd_direct_misaligned_dontcache __read_mostly;
 
 bool nfsd_v4client(struct svc_rqst *rqstp);
 
fs/nfsd/vfs.c +9 -4
index 0b39f53e041e1..1304953c684b7 100644
@@ -54,6 +54,7 @@ bool nfsd_disable_splice_read __read_mostly;
 u64 nfsd_io_cache_read __read_mostly = NFSD_IO_BUFFERED;
 u64 nfsd_io_cache_write __read_mostly = NFSD_IO_BUFFERED;
 u32 nfsd_direct_misaligned_num_pages __read_mostly = 2;
+bool nfsd_direct_misaligned_dontcache __read_mostly = true;
 
 /**
  * nfserrno - Map Linux errnos to NFS errnos
@@ -1304,10 +1305,13 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	size_t prefix, middle, suffix;
 	loff_t offset = iocb->ki_pos;
 	unsigned int dontcache_flags = 0;
+	unsigned int buffered_flags;
 	unsigned int nsegs = 0;
 
 	if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE)
 		dontcache_flags = IOCB_DONTCACHE;
+	buffered_flags = READ_ONCE(nfsd_direct_misaligned_dontcache) ?
+			 dontcache_flags : 0;
 
 	/*
 	 * Check if direct I/O is feasible for this write request.
@@ -1334,7 +1338,7 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	if (prefix) {
 		nfsd_write_dio_seg_init(&segments[nsegs], bvec,
 					nvecs, total, 0, prefix, iocb);
-		segments[nsegs++].flags |= dontcache_flags;
+		segments[nsegs++].flags |= buffered_flags;
 	}
 
 	nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs,
@@ -1355,7 +1359,7 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	if (suffix) {
 		nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
 					prefix + middle, suffix, iocb);
-		segments[nsegs++].flags |= dontcache_flags;
+		segments[nsegs++].flags |= buffered_flags;
 	}
 
 	return nsegs;
@@ -1363,11 +1367,12 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 no_dio:
 	/*
 	 * Issue the whole WRITE as a single buffered segment, uncached when
-	 * the file system supports FOP_DONTCACHE.
+	 * the file system supports FOP_DONTCACHE and
+	 * direct_misaligned_dontcache is set.
 	 */
 	nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
 				total, iocb);
-	segments[0].flags |= dontcache_flags;
+	segments[0].flags |= buffered_flags;
 	return 1;
 }
 
Signature · 1 line
2.52.0
#4 lore raw

PATCH v3 4/9 NFSD: do not use direct I/O for a READ smaller than its alignment

In reply to Mike Snitzer

nfsd_direct_read() expands a misaligned READ out to DIO-aligned
boundaries: it reads from round_down(offset, dio_read_offset_align) to round_up(offset + count, dio_read_offset_align) and returns only the requested bytes from within that window. When the READ is smaller than the alignment, that window is always at least one full alignment unit, and two when the READ straddles a boundary, so a few hundred bytes of payload can cost a 4K or 64K device read.

Decline direct I/O for those. A READ smaller than dio_read_offset_align now falls through to the DONTCACHE path, which issues DONTCACHE buffered I/O when the file system supports FOP_DONTCACHE and normal buffered I/O otherwise. This mirrors the WRITE side, which already declines direct I/O for a WRITE smaller than the larger of its offset and memory alignments.

Only dio_read_offset_align is consulted, because the READ path fills page-aligned pages from rq_bvec and so has no memory alignment to satisfy. The threshold only bites when the file system advertises a large alignment; where it reports 512 almost no READ is excluded.

Document it in the "Misaligned READ" section of nfsd-io-modes.rst.

Assisted-by Claude:claude-opus-5[1m]
---
 Documentation/filesystems/nfs/nfsd-io-modes.rst | 5 +++++
 fs/nfsd/vfs.c                                   | 6 +++---
 2 files changed, 8 insertions(+), 3 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +5 -0
index 9b1a9e7b09cef..12679001c7bec 100644
@@ -121,6 +121,11 @@ Misaligned READ:
     verified to have proper offset/len (logical_block_size) and
     dma_alignment checking.
 
+    A READ smaller than dio_read_offset_align is not expanded: it would
+    read one or two whole alignment units to return fewer bytes than
+    one. It is issued as DONTCACHE buffered IO instead (normal buffered
+    IO if the filesystem lacks FOP_DONTCACHE).
+
 Misaligned WRITE:
     If NFSD_IO_DIRECT is used, split any misaligned WRITE into a start,
     middle and end as needed. The large middle segment is DIO-aligned
fs/nfsd/vfs.c +3 -3
index 1304953c684b7..e1d294aceb6bd 100644
@@ -1187,7 +1187,7 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		      unsigned int base, u32 *eof)
 {
 	struct file *file = nf->nf_file;
-	unsigned long v, total;
+	unsigned long v, total = *count;
 	struct iov_iter iter;
 	struct kiocb kiocb;
 	ssize_t host_err;
@@ -1200,7 +1200,8 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		break;
 	case NFSD_IO_DIRECT:
 		/* When dio_read_offset_align is zero, dio is not supported */
-		if (nf->nf_dio_read_offset_align && !rqstp->rq_res.page_len)
+		if (nf->nf_dio_read_offset_align && !rqstp->rq_res.page_len &&
+		    total >= nf->nf_dio_read_offset_align)
 			return nfsd_direct_read(rqstp, fhp, nf, offset,
 						count, eof);
 		fallthrough;
@@ -1213,7 +1214,6 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	kiocb.ki_pos = offset;
 
 	v = 0;
-	total = *count;
 	while (total && v < rqstp->rq_maxpages &&
 	       rqstp->rq_next_page < rqstp->rq_page_end) {
 		len = min_t(size_t, total, PAGE_SIZE - base);
Signature · 1 line
2.52.0
#5 lore raw

PATCH v3 5/9 NFSD: persist a synchronous direct-mode WRITE once, after all of its segments

In reply to Mike Snitzer

nfsd_direct_write() may issue a WRITE as up to three segments: a buffered prefix, a direct middle and a buffered suffix. For a FILE_SYNC or DATA_SYNC WRITE the kiocb carries IOCB_DSYNC and every segment inherits it, so generic_write_sync() runs a range fsync after each segment: up to three cache flushes and log forces per WRITE, and each one writes back and drops the boundary page it just touched.

Strip IOCB_DSYNC and IOCB_SYNC from the per-segment flags and persist the WRITE once with vfs_fsync_range() over the bytes actually written, after the last segment. Durability is unchanged: the reply is not sent until the fsync completes, and datasync mirrors the previous per-segment choice (IOCB_SYNC present means metadata too). An fsync failure is returned like a write failure.

Besides the fewer flushes, this puts the sync under NFSD's control, which a later commit uses to keep the boundary pages of a split WRITE cached until the partner WRITE completes them.

Assisted-by Claude:claude-fable-5-1
---
 Documentation/filesystems/nfs/nfsd-io-modes.rst |  3 +++
 fs/nfsd/vfs.c                                   | 15 ++++++++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +3 -0
index 12679001c7bec..686b106b96f1b 100644
@@ -152,6 +152,9 @@ Misaligned WRITE:
     - the WRITE payload is not aligned in memory to the block device's
       dma_alignment, so the middle cannot be O_DIRECT either.
 
+    A FILE_SYNC or DATA_SYNC WRITE is persisted once after all of its
+    segments are written, not once per segment.
+
     Writing N to /sys/kernel/debug/nfsd/direct_misaligned_dontcache
     (default Y) issues the start and end segments, and a WRITE that is
     not split, as normal buffered IO instead of DONTCACHE, which suits a
fs/nfsd/vfs.c +14 -1
index e1d294aceb6bd..592415900403f 100644
@@ -1383,16 +1383,22 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 {
 	struct nfsd_write_dio_seg segments[3];
 	struct file *file = nf->nf_file;
+	loff_t start = kiocb->ki_pos;
+	bool sync, datasync;
 	unsigned int nsegs, i;
 	ssize_t host_err;
 	size_t expected;
 
+	/* Persist a synchronous WRITE once, after all of its segments. */
+	sync = kiocb->ki_flags & IOCB_DSYNC;
+	datasync = !(kiocb->ki_flags & IOCB_SYNC);
+
 	nsegs = nfsd_write_dio_iters_init(nf, rqstp->rq_bvec, nvecs,
 					  kiocb, *cnt, segments);
 
 	*cnt = 0;
 	for (i = 0; i < nsegs; i++) {
-		kiocb->ki_flags = segments[i].flags;
+		kiocb->ki_flags = segments[i].flags & ~(IOCB_DSYNC | IOCB_SYNC);
 		if (kiocb->ki_flags & IOCB_DIRECT)
 			trace_nfsd_write_direct(rqstp, fhp, kiocb->ki_pos,
 						segments[i].iter.count);
@@ -1410,6 +1416,13 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			break;	/* partial write */
 	}
 
+	if (sync && *cnt) {
+		host_err = vfs_fsync_range(file, start, start + *cnt - 1,
+					   datasync);
+		if (host_err < 0)
+			return host_err;
+	}
+
 	return 0;
 }
 
Signature · 1 line
2.52.0
#6 lore raw

PATCH v3 6/9 NFSD: keep boundary page of a split direct-mode WRITE until both writers complete

In reply to Mike Snitzer

A direct-mode WRITE whose ends are not aligned for direct I/O is split into a buffered prefix, a direct middle and a buffered suffix, and the prefix and suffix are issued IOCB_DONTCACHE so their pages are dropped once written back. The page holding such a segment is shared by two WRITEs, the one ending in it and the one starting in it, which may arrive in either order, from different clients, at the same time. If it is dropped as soon as the first writer's data is written back, the second has to read the block from disk before it can complete it: one 4 KiB read per WRITE. With many clients interleaving small misaligned O_DIRECT records into one shared file, that was 0.81 device reads per record (585 GiB read while writing 8.1 TiB).

Keep the page until both writers have had it, with no state in NFSD. Immediately before a boundary segment is written, nfsd_write_dio_boundary_claim() puts an empty folio in the page cache if there is not one there already. The folio is inserted unmarked: a DONTCACHE write only marks a folio it allocated itself, so neither writer marks this one and it is never counted in WB_DONTCACHE_DIRTY, which is what the DONTCACHE writeback kick targets.

The add is atomic, so of two concurrent partners exactly one gets the folio. The other finds it (-EEXIST), is therefore the second of the two, and once its data is in the page nfsd_write_dio_boundary_complete() marks the folio dropbehind. The mark must follow the write, because a clean marked folio is dropped by whatever writeback completes next, and must not be accounted, because a counted folio is handed straight back to the kick. Whichever writeback then cleans the page drops it: the WRITE's own sync for FILE_SYNC and DATA_SYNC, the flusher or the client's COMMIT for UNSTABLE.

A WRITE that is not split is issued as one buffered DONTCACHE segment. Its first and last pages are shared with the neighbouring WRITEs in the same way, and are claimed the same way. Nothing is claimed when direct_misaligned_dontcache is N, since those pages are then cached.

Measured with 32 interleaved writers over an emulated 4Kn NVMe, direct_misaligned_dontcache=Y and io_cache_write=4 (the NFSD_IO_DIRECT_WRITE_FILE_SYNC mode added in a later commit), each arm starting from a freshly made file system. 47008-byte records, which
split: 704 device reads for 42895 WRITEs, against 45144 for 45664 WRITEs without this patch. 6000-byte records, which are not split: 385 reads against 196589. The DONTCACHE flusher stays idle in both, and 81 of the written file's 524066 pages are still resident afterwards.

On a four-server pNFS flexfiles rig, three clients writing 47008-byte records for 240 s against 640 nfsd threads per server: device reads fall from 0.014-0.015 per record to 0.003, page-cache growth over the write phase falls from about 11 GiB to 2.9 GiB, and write throughput rises by 2.4% to 4.9% across NFSD_IO_DIRECT and the two modes added in a later commit, with read throughput unchanged.
Keeping every boundary page cached instead, with direct_misaligned_dontcache=N, grows the page cache by about 630 GiB over the same runs.

Assisted-by Claude:claude-opus-5[1m]
---
 .../filesystems/nfs/nfsd-io-modes.rst         | 11 +++
 fs/nfsd/vfs.c                                 | 87 ++++++++++++++++++-
 2 files changed, 94 insertions(+), 4 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +11 -0
index 686b106b96f1b..683e3a81d4de1 100644
@@ -155,6 +155,17 @@ Misaligned WRITE:
     A FILE_SYNC or DATA_SYNC WRITE is persisted once after all of its
     segments are written, not once per segment.
 
+    The page holding a start or end segment is shared with the
+    neighbouring WRITE, which may arrive before, after or at the same
+    time, from any client. So that the second WRITE to such a page does
+    not have to read it back from disk, NFSD puts an empty page there
+    immediately before writing a start or end segment, if there is not
+    one already, without marking it DONTCACHE. The WRITE that finds the
+    page already there marks it DONTCACHE once it has written it, and
+    the next writeback drops it. The first and last pages of a WRITE
+    that is not split are handled the same way. None of this applies
+    when direct_misaligned_dontcache is N.
+
     Writing N to /sys/kernel/debug/nfsd/direct_misaligned_dontcache
     (default Y) issues the start and end segments, and a WRITE that is
     not split, as normal buffered IO instead of DONTCACHE, which suits a
fs/nfsd/vfs.c +83 -4

132 lines not on this page · raw message

Signature · 1 line
2.52.0
#7 lore raw

PATCH v3 7/9 NFSD: add tracing for how direct-mode READ and WRITE are serviced

In reply to Mike Snitzer

When io_cache_read or io_cache_write selects a direct I/O mode, a
request may still be serviced with DONTCACHE buffered or normal
buffered I/O, and a WRITE may be split into up to three segments that
each take a different path depending on the request's offset/length alignment, the payload's memory alignment, the filesystem's
FOP_DONTCACHE support and direct_misaligned_dontcache. Until now the only visibility was nfsd_{read,write}_direct for O_DIRECT and nfsd_{read,write}_vector for everything else, so the DONTCACHE and cached buffered cases were indistinguishable and the reason a WRITE was not issued as direct I/O
was not recorded at all.

Add:

- nfsd_write_dio_split, emitted once per direct-mode WRITE before any segment is issued. It records the advertised offset and memory alignments, the payload's offset within its first page, the start, middle and end segment sizes, the number of segments issued, a disposition naming the path taken (direct, no_alignment, too_small, no_middle, mem_misaligned), and whether the buffered segments were issued IOCB_DONTCACHE.

- nfsd_write_dontcache and nfsd_read_dontcache, emitted for WRITE segments and READs serviced with IOCB_DONTCACHE. nfsd_write_vector and nfsd_read_vector now fire only for normal buffered I/O.

The disposition enum lives in vfs.h so trace.h can reference it.

Document the new events in nfsd-io-modes.rst, including the iomap iomap_dio_invalidate_fail event that reveals an O_DIRECT segment which the filesystem silently serviced with buffered I/O.

Assisted-by Claude:claude-fable-5-1
---
 .../filesystems/nfs/nfsd-io-modes.rst         | 34 +++++++-
 fs/nfsd/trace.h                               | 85 +++++++++++++++++++
 fs/nfsd/vfs.c                                 | 48 ++++++++---
 fs/nfsd/vfs.h                                 | 14 +++
 4 files changed, 166 insertions(+), 15 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +31 -3
index 683e3a81d4de1..11938caee14ca 100644
@@ -175,21 +175,49 @@ Misaligned WRITE:
 Tracing:
     The nfsd_read_direct trace event shows how NFSD expands any
     misaligned READ to the next DIO-aligned block (on either end of the
-    original READ, as needed).
+    original READ, as needed). A READ that is serviced with buffered IO
+    instead emits nfsd_read_dontcache (DONTCACHE buffered IO) or
+    nfsd_read_vector (normal buffered IO).
 
     This combination of trace events is useful for READs::
 
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_vector/enable
+      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_dontcache/enable
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_direct/enable
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_io_done/enable
       echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_read/enable
 
-    The nfsd_write_direct trace event shows how NFSD splits a given
-    misaligned WRITE into a DIO-aligned middle segment.
+    The nfsd_write_dio_split trace event is emitted once per WRITE in a
+    DIRECT IO mode, before any IO is issued. It records the alignments
+    the filesystem advertised, the payload's offset in its first page,
+    the sizes of the start, middle and end segments, the number of
+    segments issued, and a disposition::
+
+      direct          the aligned middle segment uses O_DIRECT
+      mem_misaligned  payload memory is misaligned; not split
+      no_alignment    the filesystem advertises no DIO alignment; not split
+      too_small       smaller than the larger alignment; not split
+      no_middle       no aligned middle, or one smaller than
+                      direct_misaligned_num_pages; not split
+
+    dontcache=1 means the buffered segments were DONTCACHE: the start and
+    end segments of a direct WRITE, or the whole of one that was not
+    split.
+
+    Each segment then emits nfsd_write_direct (O_DIRECT),
+    nfsd_write_dontcache (DONTCACHE buffered IO) or nfsd_write_vector
+    (normal buffered IO).
 
     This combination of trace events is useful for WRITEs::
 
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable
+      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dio_split/enable
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_direct/enable
+      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dontcache/enable
+      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_vector/enable
       echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable
       echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable
+      echo 1 > /sys/kernel/tracing/events/iomap/iomap_dio_invalidate_fail/enable
+
+    iomap_dio_invalidate_fail reports an O_DIRECT middle segment that
+    the filesystem serviced with buffered IO instead.
fs/nfsd/trace.h +85 -0

108 lines not on this page · raw message

fs/nfsd/vfs.c +36 -12

113 lines not on this page · raw message

fs/nfsd/vfs.h +14 -0
index 38f7d36bd4da2..4b352a48a8af2 100644
@@ -229,4 +229,18 @@ __be32		nfsd_permission(struct svc_cred *cred, struct svc_export *exp,
 
 void		nfsd_filp_close(struct file *fp);
 
+/*
+ * The path nfsd_write_dio_iters_init() took for a direct-mode WRITE.
+ * NFSD_WRITE_DIO_DONTCACHE is ORed in when its buffered segments carry
+ * IOCB_DONTCACHE.
+ */
+enum nfsd_write_dio_disposition {
+	NFSD_WRITE_DIO_DIRECT,
+	NFSD_WRITE_DIO_MEM_MISALIGNED,
+	NFSD_WRITE_DIO_NO_ALIGN,
+	NFSD_WRITE_DIO_TOO_SMALL,
+	NFSD_WRITE_DIO_NO_MIDDLE,
+	NFSD_WRITE_DIO_DONTCACHE = 0x80,
+};
+
 #endif /* LINUX_NFSD_VFS_H */
Signature · 1 line
2.52.0
#8 lore raw

PATCH v3 8/9 NFSD: Enable return of an updated stable_how to NFS clients

In reply to Mike Snitzer

From: Chuck Lever <[email protected]>

NFSv3 and newer protocols enable clients to perform a two-phase WRITE. A client requests an UNSTABLE WRITE, which sends dirty data to the NFS server, but does not persist it. The server replies that it performed the UNSTABLE WRITE, and the client is then obligated to follow up with a COMMIT request before it can remove the dirty data from its own page cache. The COMMIT reply is the client's guarantee that the written data has been persisted on the server.

The purpose of this protocol design is to enable clients to send a large amount of data via multiple WRITE requests to a server, and then wait for persistence just once. The server is able to start persisting the data as soon as it gets it, to shorten the length of time the client has to wait for the final COMMIT to complete.

It's also possible for the server to respond to an UNSTABLE WRITE request in a way that indicates that the data was persisted anyway. In that case, the client can skip the COMMIT and remove the dirty data from its memory immediately. NetApp filers, for example, do this because they have a battery-backed cache and can guarantee that written data is persisted quickly and immediately.

NFSD has never implemented this kind of promotion. UNSTABLE WRITE requests are unconditionally treated as UNSTABLE. However, in a subsequent patch, nfsd_vfs_write() will be able to promote an UNSTABLE WRITE to be a FILE_SYNC WRITE, when NFSD is configured to persist each WRITE it services with O_DIRECT before replying. The FILE_SYNC WRITE response indicates to the client that no follow-up COMMIT is necessary.

This patch prepares for that change by making the @iocb_flags argument of nfsd_write() and nfsd_vfs_write() bi-directional. A caller passes in the IOCB_* flags that express the stability its client asked for; on return the argument holds the stability to report. NFSv3 and NFSv4 convert the result back to their on-the-wire stable_how with the new nfsd3_stable_how() and nfsd4_stable_how() helpers. This keeps NFS stable_how values out of NFSD's VFS API. NFSv2 has no stable_how in its reply and ignores the result.

No behavior change is expected. In particular, a WRITE to an export with the async option is still issued without IOCB_DSYNC and still reported at the stability the client requested.

[snitzer: reworked onto the @iocb_flags argument introduced by
commit 6dcddbb70b08 ("NFSD: Replace nfsd_write()'s "stable" argument with "iocb_flags""), which was applied after this patch was first posted. The original
passed a "u32 *stable_how" instead. Carrying the value as IOCB_* flags keeps the NFSv3 XDR value out of the VFS API, and lets a later commit
raise the reported stability with a plain bitwise OR. The async
export option clears the flags on a local copy only, so the reply to
a FILE_SYNC or DATA_SYNC WRITE is never lowered (RFC 1813 section
3.3.7, RFC 8881 section 18.32.3). O_DIRECT alone does not persist
data, so the description no longer says it does. The Reviewed-by
tags from the original posting are dropped because the argument's
type and direction changed.]

---
 fs/nfsd/nfs3proc.c | 16 +++++++++++++---
 fs/nfsd/nfs4proc.c | 15 +++++++++++++--
 fs/nfsd/nfsproc.c  |  3 ++-
 fs/nfsd/vfs.c      | 17 ++++++++++-------
 fs/nfsd/vfs.h      |  4 ++--
 fs/nfsd/xdr3.h     |  2 +-
 6 files changed, 41 insertions(+), 16 deletions(-)
fs/nfsd/nfs3proc.c +13 -3
index 60cd01b6a37d2..f06573759ad6f 100644
@@ -102,6 +102,15 @@ static int nfsd3_iocb_flags(enum nfs3_stable_how how)
 	}
 }
 
+static enum nfs3_stable_how nfsd3_stable_how(int iocb_flags)
+{
+	if (iocb_flags & IOCB_SYNC)
+		return NFS_FILE_SYNC;
+	if (iocb_flags & IOCB_DSYNC)
+		return NFS_DATA_SYNC;
+	return NFS_UNSTABLE;
+}
+
 static __be32 nfsd3_map_status(__be32 status)
 {
 	switch (status) {
@@ -299,6 +308,7 @@ nfsd3_proc_write(struct svc_rqst *rqstp)
 	struct nfsd3_writeargs *argp = rqstp->rq_argp;
 	struct nfsd3_writeres *resp = rqstp->rq_resp;
 	unsigned long cnt = argp->len;
+	int iocb_flags;
 
 	dprintk("nfsd: WRITE(3)    %s %d bytes at %Lu%s\n",
 				SVCFH_fmt(&argp->fh),
@@ -312,11 +322,11 @@ nfsd3_proc_write(struct svc_rqst *rqstp)
 		return rpc_success;
 
 	fh_copy(&resp->fh, &argp->fh);
-	resp->committed = argp->stable;
+	iocb_flags = nfsd3_iocb_flags(argp->stable);
 	resp->status = nfsd_write(rqstp, &resp->fh, argp->offset,
-				  &argp->payload, &cnt,
-				  nfsd3_iocb_flags(resp->committed),
+				  &argp->payload, &cnt, &iocb_flags,
 				  resp->verf);
+	resp->committed = nfsd3_stable_how(iocb_flags);
 	resp->count = cnt;
 	resp->status = nfsd3_map_status(resp->status);
 	return rpc_success;
fs/nfsd/nfs4proc.c +13 -2
index 7df60abfbff14..505c7dadf49b8 100644
@@ -109,6 +109,15 @@ static const struct nfsd_access_maps nfsd4_access_maps = {
 	.other		= nfsd4_otheraccess,
 };
 
+static enum stable_how4 nfsd4_stable_how(int iocb_flags)
+{
+	if (iocb_flags & IOCB_SYNC)
+		return FILE_SYNC4;
+	if (iocb_flags & IOCB_DSYNC)
+		return DATA_SYNC4;
+	return UNSTABLE4;
+}
+
 static int nfsd4_iocb_flags(enum stable_how4 how)
 {
 	switch (how) {
@@ -1442,6 +1451,7 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfsd_file *nf = NULL;
 	__be32 status = nfs_ok;
 	unsigned long cnt;
+	int iocb_flags;
 
 	if (write->wr_offset > (u64)OFFSET_MAX ||
 	    write->wr_offset + write->wr_buflen > (u64)OFFSET_MAX)
@@ -1460,11 +1470,12 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		nfs4_put_stid(stid);
 	}
 
-	write->wr_how_written = write->wr_stable_how;
+	iocb_flags = nfsd4_iocb_flags(write->wr_stable_how);
 	status = nfsd_vfs_write(rqstp, &cstate->current_fh, nf,
 				write->wr_offset, &write->wr_payload,
-				&cnt, nfsd4_iocb_flags(write->wr_how_written),
+				&cnt, &iocb_flags,
 				(__be32 *)write->wr_verifier.data);
+	write->wr_how_written = nfsd4_stable_how(iocb_flags);
 	nfsd_file_put(nf);
 
 	write->wr_bytes_written = cnt;
fs/nfsd/nfsproc.c +2 -1
index 1f74008bc06ad..6565d301c7e31 100644
@@ -644,12 +644,13 @@ static __be32 nfsd_proc_write(struct svc_rqst *rqstp)
 	struct kstat *statp = &resp->stat;
 	unsigned long count = argp->xdrgen.data.len;
 	struct svc_fh *fhp = &argp->fh;
+	int iocb_flags = IOCB_DSYNC;
 
 	nfsd_fhandle_to_svc_fh(fhp, &argp->xdrgen.file);
 
 	resp->xdrgen.status = nfsd_write(rqstp, fhp, argp->xdrgen.offset,
 					 &argp->xdrgen.data, &count,
-					 IOCB_DSYNC, NULL);
+					 &iocb_flags, NULL);
 	if (resp->xdrgen.status == nfs_ok) {
 		resp->xdrgen.status = fh_getattr(fhp, statp);
 		if (resp->xdrgen.status == nfs_ok)
fs/nfsd/vfs.c +10 -7
index ec52c67c4380d..a8e15193b2160 100644
@@ -1537,7 +1537,8 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
  * @offset: Byte offset of start
  * @payload: xdr_buf containing the write payload
  * @cnt: IN: number of bytes to write, OUT: number of bytes actually written
- * @iocb_flags: VFS IOCB_* flags expressing the requested write stability
+ * @iocb_flags: IN: VFS IOCB_* flags expressing the requested write
+ *             stability; OUT: the stability to report to the client
  * @verf: NFS WRITE verifier
  *
  * Upon return, caller must invoke fh_put on @fhp.
@@ -1549,11 +1550,12 @@ __be32
 nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	       struct nfsd_file *nf, loff_t offset,
 	       const struct xdr_buf *payload, unsigned long *cnt,
-	       int iocb_flags, __be32 *verf)
+	       int *iocb_flags, __be32 *verf)
 {
 	struct nfsd_net		*nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
 	struct file		*file = nf->nf_file;
 	struct super_block	*sb = file_inode(file)->i_sb;
+	int			stable_flags = *iocb_flags;
 	struct kiocb		kiocb;
 	struct svc_export	*exp;
 	struct iov_iter		iter;
@@ -1586,11 +1588,11 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	exp = fhp->fh_export;
 
 	if (!EX_ISSYNC(exp))
-		iocb_flags = 0;
+		stable_flags = 0;
 	init_sync_kiocb(&kiocb, file);
 	kiocb.ki_pos = offset;
 	if (likely(!fhp->fh_use_wgather))
-		kiocb.ki_flags |= iocb_flags;
+		kiocb.ki_flags |= stable_flags;
 
 	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
 	if (nvecs < 0) {
@@ -1631,7 +1633,7 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		goto out_nfserr;
 	}
 
-	if (iocb_flags && fhp->fh_use_wgather) {
+	if (stable_flags && fhp->fh_use_wgather) {
 		host_err = wait_for_concurrent_writes(file);
 		if (host_err < 0)
 			nfsd_maybe_reset_write_verifier(nn, rqstp, host_err);
@@ -1722,7 +1724,8 @@ __be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
  * @offset: Byte offset of start
  * @payload: xdr_buf containing the write payload
  * @cnt: IN: number of bytes to write, OUT: number of bytes actually written
- * @iocb_flags: VFS IOCB_* flags expressing the requested write stability
+ * @iocb_flags: IN: VFS IOCB_* flags expressing the requested write
+ *             stability; OUT: the stability to report to the client
  * @verf: NFS WRITE verifier
  *
  * Upon return, caller must invoke fh_put on @fhp.
@@ -1733,7 +1736,7 @@ __be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 __be32
 nfsd_write(struct svc_rqst *rqstp, struct svc_fh *fhp, loff_t offset,
 	   const struct xdr_buf *payload, unsigned long *cnt,
-	   int iocb_flags, __be32 *verf)
+	   int *iocb_flags, __be32 *verf)
 {
 	struct nfsd_file *nf;
 	__be32 err;
fs/nfsd/vfs.h +2 -2
index 4b352a48a8af2..c3b48c45e10a7 100644
@@ -189,12 +189,12 @@ __be32		nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 				u32 *eof);
 __be32		nfsd_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 				loff_t offset, const struct xdr_buf *payload,
-				unsigned long *cnt, int iocb_flags,
+				unsigned long *cnt, int *iocb_flags,
 				__be32 *verf);
 __be32		nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 				struct nfsd_file *nf, loff_t offset,
 				const struct xdr_buf *payload,
-				unsigned long *cnt, int iocb_flags,
+				unsigned long *cnt, int *iocb_flags,
 				__be32 *verf);
 __be32		nfsd_readlink(struct svc_rqst *, struct svc_fh *,
 				char *, int *);
fs/nfsd/xdr3.h +1 -1
index 22272695e451b..eb9d79e57fc9a 100644
@@ -154,7 +154,7 @@ struct nfsd3_writeres {
 	__be32			status;
 	struct svc_fh		fh;
 	unsigned long		count;
-	int			committed;
+	u32			committed;
 	__be32			verf[2];
 };
 
Signature · 1 line
2.52.0
#9 lore raw

PATCH v3 9/9 NFSD: add direct-mode WRITE settings that persist each WRITE

In reply to Mike Snitzer

Under NFSD_IO_DIRECT an UNSTABLE WRITE is issued O_DIRECT without IOCB_DSYNC and answered UNSTABLE. The data is not yet durable: the device cache is not flushed, and block allocation, unwritten extent conversion and the size update are not committed. The client's COMMIT is what makes it so. This patch does not change NFSD_IO_DIRECT.

Add two io_cache_write modes that issue direct I/O like NFSD_IO_DIRECT but persist every WRITE before replying, to at least a floor, and report the stability achieved:

NFSD_IO_DIRECT_WRITE_DATA_SYNC (3): at least NFS_DATA_SYNC NFSD_IO_DIRECT_WRITE_FILE_SYNC (4): at least NFS_FILE_SYNC

A client that asked for more is left alone. A client told FILE_SYNC sends no COMMIT.

The cost is a flush and log force on every WRITE, where NFSD_IO_DIRECT has one per COMMIT, so these are opt-in. They cost more than NFSD_IO_DIRECT when one COMMIT covers many WRITEs. They cost the same flushes, and save the COMMIT RPCs, when each COMMIT covers about one WRITE. That is the case on a pNFS flexfiles share where a client's writes straddle two data servers, so that each data server receives one WRITE and one COMMIT per write. Measured there, with identical WRITE counts and flush counts, the COMMIT RPCs cost NFSD_IO_DIRECT 31% more server CPU and 42% more client CPU for the same bytes. Where only one write in 22 was split, there was no resolvable difference.

A Linux client change that sends FILE_SYNC for a WRITE that is alone in its flush to a given server would remove those COMMITs without a server setting, for clients that have it.

Mode 3 promises only the data, leaving a client that needs metadata durability to COMMIT for it. Both modes are inert for READ and for a WRITE the client already marked FILE_SYNC.

Assisted-by Claude:claude-opus-5[1m]
---
 .../filesystems/nfs/nfsd-io-modes.rst         | 15 ++++++--
 fs/nfsd/debugfs.c                             |  5 +++
 fs/nfsd/nfsd.h                                |  2 ++
 fs/nfsd/vfs.c                                 | 35 ++++++++++++++++---
 4 files changed, 50 insertions(+), 7 deletions(-)
Documentation/filesystems/nfs/nfsd-io-modes.rst +13 -2
index 11938caee14ca..87266e43e4d1c 100644
@@ -25,12 +25,14 @@ Based on the configured settings, NFSD's IO will either be:
 - cached using page cache (NFSD_IO_BUFFERED=0)
 - cached but removed from page cache on completion (NFSD_IO_DONTCACHE=1)
 - not cached stable_how=NFS_UNSTABLE (NFSD_IO_DIRECT=2)
+- not cached stable_how=NFS_DATA_SYNC (NFSD_IO_DIRECT_WRITE_DATA_SYNC=3)
+- not cached stable_how=NFS_FILE_SYNC (NFSD_IO_DIRECT_WRITE_FILE_SYNC=4)
 
-To set an NFSD IO mode, write a supported value (0 - 2) to the
+To set an NFSD IO mode, write a supported value (0 - 4) to the
 corresponding IO operation's debugfs interface, e.g.::
 
   echo 2 > /sys/kernel/debug/nfsd/io_cache_read
-  echo 2 > /sys/kernel/debug/nfsd/io_cache_write
+  echo 4 > /sys/kernel/debug/nfsd/io_cache_write
 
 To check which IO mode NFSD is using for READ or WRITE, simply read the
 corresponding IO operation's debugfs interface, e.g.::
@@ -38,6 +40,15 @@ corresponding IO operation's debugfs interface, e.g.::
   cat /sys/kernel/debug/nfsd/io_cache_read
   cat /sys/kernel/debug/nfsd/io_cache_write
 
+NFSD_IO_DIRECT leaves an UNSTABLE WRITE for the client's COMMIT to
+persist. The two NFSD_IO_DIRECT_WRITE_*_SYNC modes instead persist every
+WRITE before replying, to at least NFS_DATA_SYNC or NFS_FILE_SYNC, and
+report that stable_how to the client; a client that asked for more is
+left alone. With NFSD_IO_DIRECT_WRITE_FILE_SYNC the client sends no
+COMMIT. This costs a flush on every WRITE: more than NFSD_IO_DIRECT
+when one COMMIT covers many WRITEs, about the same, without the COMMIT
+RPCs, when each COMMIT covers about one.
+
 If you experiment with NFSD's IO modes on a recent kernel and have
 interesting results, please report them to [email protected]
 
fs/nfsd/debugfs.c +5 -0
index d5b714dd0fbc6..bd8ceb5eb2879 100644
@@ -90,6 +90,9 @@ DEFINE_DEBUGFS_ATTRIBUTE(nfsd_io_cache_read_fops, nfsd_io_cache_read_get,
  * Contents:
  *   %0: NFS WRITE will use buffered IO
  *   %1: NFS WRITE will use dontcache (buffered IO w/ dropbehind)
+ *   %2: NFS WRITE will use direct IO with stable_how=NFS_UNSTABLE
+ *   %3: NFS WRITE will use direct IO with stable_how=NFS_DATA_SYNC
+ *   %4: NFS WRITE will use direct IO with stable_how=NFS_FILE_SYNC
  *
  * This setting takes immediate effect for all NFS versions,
  * all exports, and in all NFSD net namespaces.
@@ -109,6 +112,8 @@ static int nfsd_io_cache_write_set(void *data, u64 val)
 	case NFSD_IO_BUFFERED:
 	case NFSD_IO_DONTCACHE:
 	case NFSD_IO_DIRECT:
+	case NFSD_IO_DIRECT_WRITE_DATA_SYNC:
+	case NFSD_IO_DIRECT_WRITE_FILE_SYNC:
 		nfsd_io_cache_write = val;
 		break;
 	default:
fs/nfsd/nfsd.h +2 -0
index 1c17f96012523..913860ab3eb20 100644
@@ -141,6 +141,8 @@ enum {
 	NFSD_IO_BUFFERED,
 	NFSD_IO_DONTCACHE,
 	NFSD_IO_DIRECT,
+	NFSD_IO_DIRECT_WRITE_DATA_SYNC,
+	NFSD_IO_DIRECT_WRITE_FILE_SYNC,
 };
 
 extern u64 nfsd_io_cache_read __read_mostly;
fs/nfsd/vfs.c +30 -5
index a8e15193b2160..af05ef942c5d9 100644
@@ -1450,12 +1450,25 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	return nsegs;
 }
 
+/* Raise this WRITE to at least @floor_iocb_flags, and report that. */
+static void
+nfsd_write_raise_stability(int floor_iocb_flags, struct kiocb *kiocb,
+			   int *iocb_flags)
+{
+	if ((*iocb_flags & floor_iocb_flags) == floor_iocb_flags)
+		return;
+
+	*iocb_flags |= floor_iocb_flags;
+	kiocb->ki_flags |= floor_iocb_flags;
+}
+
 static noinline_for_stack int
 nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
-		  struct nfsd_file *nf, unsigned int nvecs,
+		  struct nfsd_file *nf, int *iocb_flags, unsigned int nvecs,
 		  unsigned long *cnt, struct kiocb *kiocb)
 {
 	struct nfsd_write_dio_seg segments[3];
+	int floor_iocb_flags = 0;
 	struct file *file = nf->nf_file;
 	loff_t start = kiocb->ki_pos, seg_pos, seg_last;
 	bool sync, datasync, complete_first, complete_last;
@@ -1463,6 +1476,14 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	ssize_t host_err;
 	size_t expected;
 
+	if (nfsd_io_cache_write == NFSD_IO_DIRECT_WRITE_FILE_SYNC)
+		floor_iocb_flags = IOCB_DSYNC | IOCB_SYNC;
+	else if (nfsd_io_cache_write == NFSD_IO_DIRECT_WRITE_DATA_SYNC)
+		floor_iocb_flags = IOCB_DSYNC;
+	if (floor_iocb_flags)
+		nfsd_write_raise_stability(floor_iocb_flags, kiocb,
+					   iocb_flags);
+
 	/* Persist a synchronous WRITE once, after all of its segments. */
 	sync = kiocb->ki_flags & IOCB_DSYNC;
 	datasync = !(kiocb->ki_flags & IOCB_SYNC);
@@ -1538,7 +1559,8 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
  * @payload: xdr_buf containing the write payload
  * @cnt: IN: number of bytes to write, OUT: number of bytes actually written
  * @iocb_flags: IN: VFS IOCB_* flags expressing the requested write
- *             stability; OUT: the stability to report to the client
+ *             stability; OUT: the stability to report to the client,
+ *             which may be raised
  * @verf: NFS WRITE verifier
  *
  * Upon return, caller must invoke fh_put on @fhp.
@@ -1606,8 +1628,10 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 
 	switch (nfsd_io_cache_write) {
 	case NFSD_IO_DIRECT:
-		host_err = nfsd_direct_write(rqstp, fhp, nf, nvecs,
-					     cnt, &kiocb);
+	case NFSD_IO_DIRECT_WRITE_DATA_SYNC:
+	case NFSD_IO_DIRECT_WRITE_FILE_SYNC:
+		host_err = nfsd_direct_write(rqstp, fhp, nf, iocb_flags,
+					     nvecs, cnt, &kiocb);
 		break;
 	case NFSD_IO_DONTCACHE:
 		if (file->f_op->fop_flags & FOP_DONTCACHE)
@@ -1725,7 +1749,8 @@ __be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
  * @payload: xdr_buf containing the write payload
  * @cnt: IN: number of bytes to write, OUT: number of bytes actually written
  * @iocb_flags: IN: VFS IOCB_* flags expressing the requested write
- *             stability; OUT: the stability to report to the client
+ *             stability; OUT: the stability to report to the client,
+ *             which may be raised
  * @verf: NFS WRITE verifier
  *
  * Upon return, caller must invoke fh_put on @fhp.
Signature · 1 line
2.52.0
#10 lore raw
In reply to Mike Snitzer

Just FYI, while this v3 is vastly improved (thanks to it addressng most of Chuck's detailed review of v2) it still has issues that I'm addressing.

I'll post v4 when its ready, so please don't waste time reviewing v3.

Thanks,
Mike