Skip to contentSkip to conversation

NFSD DIRECT: misaligned READs fixup, add handling for misaligned WRITEs

PATCH In review Mike Snitzer 2 participants 13 messages Aug 5, 2025 – Aug 7, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover NFSD DIRECT: misaligned READs fixup, add handling for misaligned WRITEs

Hi,

This series builds on what has been staged in the nfsd-testing branch.

This code has proven to work well during my testing. Any suggestions for further refinement are welcome.

Changes since v3:
- fixed nfsd_vfs_write() so work that only needs to happen once, after IO is submitted, isn't performed each iteration of the for loop, thanks for catching this Chuck.
- updated NFSD's misaligned READ and WRITE handling to not use iov_iter_is_aligned() because it will soon be removed upstream.
- Chuck, provided both you and Jeff agree with patch 1's incremental changes, patch 1 should be folded into what you already have in your nfsd-testing branch (more justification in patch 1's header)
- dropped the NFSD misaligned DIO NFS reexport patch in favor of adding STATX_DIOALIGN support to NFS (the patch to add support will be provided in the next NFS DIRECT v7 patchset that I'll post soon).

Changes since v2:
- fixed patch 2 by moving redundant work out of nfsd_vfs_write()'s for loop, thanks to Jeff's review.
- added Jeff's Reviewed-by to patches 1-3.
- Left patch 4 in the series because it is pragmatic, but feel free to drop it if you'd prefer to see this cat skinned a different way.

Changes since v1:
- switched to using an EVENT_CLASS to create nfsd_analyze_{read,write}_dio
- added 4th patch, if user configured use of NFSD_IO_DIRECT then NFS reexports should use it too (in future, with per-export controls we'll have the benefit of finer-grained control; but until then we'd do well to offer comprehensive use of NFSD_IO_DIRECT if it enabled).

Thanks,
Mike

Mike Snitzer (4):
NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()
NFSD: refactor nfsd_read_vector_dio to EVENT_CLASS useful for READ and WRITE
NFSD: prepare nfsd_vfs_write() to use O_DIRECT on misaligned WRITEs
NFSD: issue WRITEs using O_DIRECT even if IO is misaligned

 fs/nfsd/trace.h |  52 +++++++++----
 fs/nfsd/vfs.c   | 199 +++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 202 insertions(+), 49 deletions(-)
Signature · 1 line
2.44.0
#1 lore raw

PATCH v4 1/4 NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()

From: Mike Snitzer <[email protected]>

Check the bvec is DIO-aligned while creating it, saves CPU cycles by avoiding iterating the bvec elements a second time using iov_iter_is_aligned().

This prepares for Keith Busch's near-term removal of the iov_iter_is_aligned() interface. This fixes cel/nfsd-testing commit 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned") and it should be folded into that commit so that NFSD doesn't require iov_iter_is_aligned() while it is being removed upstream in parallel.

Fixes cel/nfsd-testing 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned")
---
 fs/nfsd/vfs.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
fs/nfsd/vfs.c +15 -14
index 46189020172fb..e1751d3715264 100644
@@ -1226,7 +1226,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			 */
 			offset = read_dio.start;
 			in_count = read_dio.end - offset;
-			kiocb.ki_flags = IOCB_DIRECT;
+			/* Verify ondisk DIO alignment, memory addrs checked below */
+			if (likely(((offset | in_count) &
+				    (nf->nf_dio_read_offset_align - 1)) == 0))
+				kiocb.ki_flags = IOCB_DIRECT;
 		}
 	} else if (nfsd_io_cache_read == NFSD_IO_DONTCACHE)
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1236,16 +1239,24 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	v = 0;
 	total = in_count;
 	if (read_dio.start_extra) {
-		bvec_set_page(&rqstp->rq_bvec[v++], read_dio.start_extra_page,
+		bvec_set_page(&rqstp->rq_bvec[v], read_dio.start_extra_page,
 			      read_dio.start_extra, PAGE_SIZE - read_dio.start_extra);
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&
+			     rqstp->rq_bvec[v].bv_offset & (nf->nf_dio_mem_align - 1)))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= read_dio.start_extra;
+		v++;
 	}
 	while (total) {
 		len = min_t(size_t, total, PAGE_SIZE - base);
-		bvec_set_page(&rqstp->rq_bvec[v++], *(rqstp->rq_next_page++),
-			      len, base);
+		bvec_set_page(&rqstp->rq_bvec[v], *(rqstp->rq_next_page++), len, base);
+		/* No need to verify memory is DIO-aligned since bv_offset is 0 */
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) && base &&
+			     (base & (nf->nf_dio_mem_align - 1))))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= len;
 		base = 0;
+		v++;
 	}
 	if (WARN_ONCE(v > rqstp->rq_maxpages,
 		      "%s: v=%lu exceeds rqstp->rq_maxpages=%lu\n", __func__,
@@ -1256,16 +1267,6 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!host_err) {
 		trace_nfsd_read_vector(rqstp, fhp, offset, in_count);
 		iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, in_count);
-
-		/* Double check nfsd_analyze_read_dio's DIO-aligned result */
-		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&
-			     !iov_iter_is_aligned(&iter,
-				nf->nf_dio_mem_align - 1,
-				nf->nf_dio_read_offset_align - 1))) {
-			/* Fallback to buffered IO */
-			kiocb.ki_flags &= ~IOCB_DIRECT;
-		}
-
 		host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
 	}
 
Signature · 1 line
2.44.0
#2 lore raw

Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()

Quoted 70 lines · Mike Snitzer

From: Mike Snitzer <[email protected]>

Check the bvec is DIO-aligned while creating it, saves CPU cycles by avoiding iterating the bvec elements a second time using iov_iter_is_aligned().

This prepares for Keith Busch's near-term removal of the iov_iter_is_aligned() interface. This fixes cel/nfsd-testing commit 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned") and it should be folded into that commit so that NFSD doesn't require iov_iter_is_aligned() while it is being removed upstream in parallel.

Fixes cel/nfsd-testing 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned")
---
 fs/nfsd/vfs.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
fs/nfsd/vfs.c +15 -7
index 46189020172fb..e1751d3715264 100644
@@ -1226,7 +1226,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			 */
 			offset = read_dio.start;
 			in_count = read_dio.end - offset;
-			kiocb.ki_flags = IOCB_DIRECT;
+			/* Verify ondisk DIO alignment, memory addrs checked below */
+			if (likely(((offset | in_count) &
+				    (nf->nf_dio_read_offset_align - 1)) == 0))
+				kiocb.ki_flags = IOCB_DIRECT;
 		}
 	} else if (nfsd_io_cache_read == NFSD_IO_DONTCACHE)
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1236,16 +1239,24 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	v = 0;
 	total = in_count;
 	if (read_dio.start_extra) {
-		bvec_set_page(&rqstp->rq_bvec[v++], read_dio.start_extra_page,
+		bvec_set_page(&rqstp->rq_bvec[v], read_dio.start_extra_page,
 			      read_dio.start_extra, PAGE_SIZE - read_dio.start_extra);
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&
+			     rqstp->rq_bvec[v].bv_offset & (nf->nf_dio_mem_align - 1)))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= read_dio.start_extra;
+		v++;
 	}
 	while (total) {
 		len = min_t(size_t, total, PAGE_SIZE - base);
-		bvec_set_page(&rqstp->rq_bvec[v++], *(rqstp->rq_next_page++),
-			      len, base);
+		bvec_set_page(&rqstp->rq_bvec[v], *(rqstp->rq_next_page++), len, base);
+		/* No need to verify memory is DIO-aligned since bv_offset is 0 */
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) && base &&
+			     (base & (nf->nf_dio_mem_align - 1))))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= len;
 		base = 0;
+		v++;
 	}
 	if (WARN_ONCE(v > rqstp->rq_maxpages,
 		      "%s: v=%lu exceeds rqstp->rq_maxpages=%lu\n", __func__,
@@ -1256,16 +1267,6 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!host_err) {
 		trace_nfsd_read_vector(rqstp, fhp, offset, in_count);
 		iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, in_count);
-
-		/* Double check nfsd_analyze_read_dio's DIO-aligned result */
-		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&

fs/nfsd/vfs.c __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1272

Quoted 10 lines
fs/nfsd/vfs.c +0 -7
-			     !iov_iter_is_aligned(&iter,
-				nf->nf_dio_mem_align - 1,
-				nf->nf_dio_read_offset_align - 1))) {
-			/* Fallback to buffered IO */
-			kiocb.ki_flags &= ~IOCB_DIRECT;
-		}
-
 		host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
 	}
 

Hi Mike,

In cases where the SQUASHME patch is this large, I usually drop the patch (or series) in nfsd-testing and ask the contributor to rebase and repost. This gets the new version of the patch properly archived on lore, for one thing.

Before reposting, please do run checkpatch.pl on the series.

Signature · 1 line
Chuck Lever
#3 lore raw

Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()

Quoted 84 lines · Chuck Lever
Quoted 70 lines · Mike Snitzer

From: Mike Snitzer <[email protected]>

Check the bvec is DIO-aligned while creating it, saves CPU cycles by avoiding iterating the bvec elements a second time using iov_iter_is_aligned().

This prepares for Keith Busch's near-term removal of the iov_iter_is_aligned() interface. This fixes cel/nfsd-testing commit 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned") and it should be folded into that commit so that NFSD doesn't require iov_iter_is_aligned() while it is being removed upstream in parallel.

Fixes cel/nfsd-testing 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned")
---
 fs/nfsd/vfs.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
fs/nfsd/vfs.c +15 -7
index 46189020172fb..e1751d3715264 100644
@@ -1226,7 +1226,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			 */
 			offset = read_dio.start;
 			in_count = read_dio.end - offset;
-			kiocb.ki_flags = IOCB_DIRECT;
+			/* Verify ondisk DIO alignment, memory addrs checked below */
+			if (likely(((offset | in_count) &
+				    (nf->nf_dio_read_offset_align - 1)) == 0))
+				kiocb.ki_flags = IOCB_DIRECT;
 		}
 	} else if (nfsd_io_cache_read == NFSD_IO_DONTCACHE)
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1236,16 +1239,24 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	v = 0;
 	total = in_count;
 	if (read_dio.start_extra) {
-		bvec_set_page(&rqstp->rq_bvec[v++], read_dio.start_extra_page,
+		bvec_set_page(&rqstp->rq_bvec[v], read_dio.start_extra_page,
 			      read_dio.start_extra, PAGE_SIZE - read_dio.start_extra);
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&
+			     rqstp->rq_bvec[v].bv_offset & (nf->nf_dio_mem_align - 1)))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= read_dio.start_extra;
+		v++;
 	}
 	while (total) {
 		len = min_t(size_t, total, PAGE_SIZE - base);
-		bvec_set_page(&rqstp->rq_bvec[v++], *(rqstp->rq_next_page++),
-			      len, base);
+		bvec_set_page(&rqstp->rq_bvec[v], *(rqstp->rq_next_page++), len, base);
+		/* No need to verify memory is DIO-aligned since bv_offset is 0 */
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) && base &&
+			     (base & (nf->nf_dio_mem_align - 1))))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= len;
 		base = 0;
+		v++;
 	}
 	if (WARN_ONCE(v > rqstp->rq_maxpages,
 		      "%s: v=%lu exceeds rqstp->rq_maxpages=%lu\n", __func__,
@@ -1256,16 +1267,6 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!host_err) {
 		trace_nfsd_read_vector(rqstp, fhp, offset, in_count);
 		iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, in_count);
-
-		/* Double check nfsd_analyze_read_dio's DIO-aligned result */
-		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&

fs/nfsd/vfs.c __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1272

Quoted 10 lines
fs/nfsd/vfs.c +0 -7
-			     !iov_iter_is_aligned(&iter,
-				nf->nf_dio_mem_align - 1,
-				nf->nf_dio_read_offset_align - 1))) {
-			/* Fallback to buffered IO */
-			kiocb.ki_flags &= ~IOCB_DIRECT;
-		}
-
 		host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
 	}
 

Hi Mike,

Quoted 4 lines

In cases where the SQUASHME patch is this large, I usually drop the patch (or series) in nfsd-testing and ask the contributor to rebase and repost. This gets the new version of the patch properly archived on lore, for one thing.

Yeah, make sense, I missed that iov_iter_is_aligned() was used early on in the series too, so I'll fixup further back.

Quoted 1 line

Before reposting, please do run checkpatch.pl on the series.

Will do, will also ensure bisect safe and that sparse is happy.

Thanks,
Mike

#4 lore raw

Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()

Quoted 90 lines · Mike Snitzer
Quoted 84 lines · Chuck Lever
Quoted 70 lines · Mike Snitzer

From: Mike Snitzer <[email protected]>

Check the bvec is DIO-aligned while creating it, saves CPU cycles by avoiding iterating the bvec elements a second time using iov_iter_is_aligned().

This prepares for Keith Busch's near-term removal of the iov_iter_is_aligned() interface. This fixes cel/nfsd-testing commit 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned") and it should be folded into that commit so that NFSD doesn't require iov_iter_is_aligned() while it is being removed upstream in parallel.

Fixes cel/nfsd-testing 5d78ac1e674b4 ("NFSD: issue READs using O_DIRECT even if IO is misaligned")
---
 fs/nfsd/vfs.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
fs/nfsd/vfs.c +15 -7
index 46189020172fb..e1751d3715264 100644
@@ -1226,7 +1226,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			 */
 			offset = read_dio.start;
 			in_count = read_dio.end - offset;
-			kiocb.ki_flags = IOCB_DIRECT;
+			/* Verify ondisk DIO alignment, memory addrs checked below */
+			if (likely(((offset | in_count) &
+				    (nf->nf_dio_read_offset_align - 1)) == 0))
+				kiocb.ki_flags = IOCB_DIRECT;
 		}
 	} else if (nfsd_io_cache_read == NFSD_IO_DONTCACHE)
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1236,16 +1239,24 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	v = 0;
 	total = in_count;
 	if (read_dio.start_extra) {
-		bvec_set_page(&rqstp->rq_bvec[v++], read_dio.start_extra_page,
+		bvec_set_page(&rqstp->rq_bvec[v], read_dio.start_extra_page,
 			      read_dio.start_extra, PAGE_SIZE - read_dio.start_extra);
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&
+			     rqstp->rq_bvec[v].bv_offset & (nf->nf_dio_mem_align - 1)))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= read_dio.start_extra;
+		v++;
 	}
 	while (total) {
 		len = min_t(size_t, total, PAGE_SIZE - base);
-		bvec_set_page(&rqstp->rq_bvec[v++], *(rqstp->rq_next_page++),
-			      len, base);
+		bvec_set_page(&rqstp->rq_bvec[v], *(rqstp->rq_next_page++), len, base);
+		/* No need to verify memory is DIO-aligned since bv_offset is 0 */
+		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) && base &&
+			     (base & (nf->nf_dio_mem_align - 1))))
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		total -= len;
 		base = 0;
+		v++;
 	}
 	if (WARN_ONCE(v > rqstp->rq_maxpages,
 		      "%s: v=%lu exceeds rqstp->rq_maxpages=%lu\n", __func__,
@@ -1256,16 +1267,6 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!host_err) {
 		trace_nfsd_read_vector(rqstp, fhp, offset, in_count);
 		iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, in_count);
-
-		/* Double check nfsd_analyze_read_dio's DIO-aligned result */
-		if (unlikely((kiocb.ki_flags & IOCB_DIRECT) &&

fs/nfsd/vfs.c __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1272

Quoted 10 lines
fs/nfsd/vfs.c +0 -7
-			     !iov_iter_is_aligned(&iter,
-				nf->nf_dio_mem_align - 1,
-				nf->nf_dio_read_offset_align - 1))) {
-			/* Fallback to buffered IO */
-			kiocb.ki_flags &= ~IOCB_DIRECT;
-		}
-
 		host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
 	}
 

Hi Mike,

Quoted 4 lines

In cases where the SQUASHME patch is this large, I usually drop the patch (or series) in nfsd-testing and ask the contributor to rebase and repost. This gets the new version of the patch properly archived on lore, for one thing.

Quoted 6 lines

Yeah, make sense, I missed that iov_iter_is_aligned() was used early on in the series too, so I'll fixup further back.

Quoted 1 line

Before reposting, please do run checkpatch.pl on the series.

Will do, will also ensure bisect safe and that sparse is happy.

Thanks for the changes, sounds like the right path forward.

Signature · 1 line
Chuck Lever
#5 lore raw

sparse warnings with nfsd-testing [was: Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()]

In reply to Mike Snitzer
Quoted 5 lines · Mike Snitzer
Quoted 1 line · Chuck Lever

Before reposting, please do run checkpatch.pl on the series.

Will do, will also ensure bisect safe and that sparse is happy.

FYI, I'm preparing my next patchset and sparse is happy with it, but I wanted to share these warnings seen with nfsd-testing through commit ae83299cc048e ("NFSD: Fix last write offset handling in layoutcommit"):

fs/nfsd/nfs4state.c: note: in included file (through include/linux/wait.h, include/linux/wait_bit.h, include/linux/fs.h):
./include/linux/list.h:229:25: warning: context imbalance in 'put_clnt_odstate' - unexpected unlock fs/nfsd/nfs4state.c:1188:9: warning: context imbalance in 'nfs4_put_stid' - unexpected unlock

I haven't looked at them closer. Could be you're well aware of them?

Mike

ps. but full disclosure, my baseline kernel is 6.12.24, I haven't tried sparse against the nfsd-testing branch itself (which is based on your nfsd-6.17); but my 6.12.24 kernel does have all NFS/NFSD/SUNRPC changes through nfsd-6.17 + nfsd-testing commit ae83299cc048e).

#6 lore raw

Re: sparse warnings with nfsd-testing [was: Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()]

Quoted 12 lines · Mike Snitzer
Quoted 5 lines · Mike Snitzer
Quoted 1 line · Chuck Lever

Before reposting, please do run checkpatch.pl on the series.

Will do, will also ensure bisect safe and that sparse is happy.

FYI, I'm preparing my next patchset and sparse is happy with it, but I wanted to share these warnings seen with nfsd-testing through commit ae83299cc048e ("NFSD: Fix last write offset handling in layoutcommit"):

Quoted 5 lines

fs/nfsd/nfs4state.c: note: in included file (through include/linux/wait.h, include/linux/wait_bit.h, include/linux/fs.h):
./include/linux/list.h:229:25: warning: context imbalance in 'put_clnt_odstate' - unexpected unlock fs/nfsd/nfs4state.c:1188:9: warning: context imbalance in 'nfs4_put_stid' - unexpected unlock

I haven't looked at them closer. Could be you're well aware of them?

These warnings have been there forever. I'm told they are the result of bugs in sparse.

Quoted 4 lines

ps. but full disclosure, my baseline kernel is 6.12.24, I haven't tried sparse against the nfsd-testing branch itself (which is based on your nfsd-6.17); but my 6.12.24 kernel does have all NFS/NFSD/SUNRPC changes through nfsd-6.17 + nfsd-testing commit ae83299cc048e).

Signature · 1 line
Chuck Lever
#7 lore raw

Re: sparse warnings with nfsd-testing [was: Re: [PATCH v4 1/4] NFSD: avoid using iov_iter_is_aligned() in nfsd_iter_read()]

Quoted 13 lines · Chuck Lever
Quoted 12 lines · Mike Snitzer
Quoted 5 lines · Mike Snitzer
Quoted 1 line · Chuck Lever

Before reposting, please do run checkpatch.pl on the series.

Will do, will also ensure bisect safe and that sparse is happy.

FYI, I'm preparing my next patchset and sparse is happy with it, but I wanted to share these warnings seen with nfsd-testing through commit ae83299cc048e ("NFSD: Fix last write offset handling in layoutcommit"):

Quoted 8 lines
Quoted 5 lines

fs/nfsd/nfs4state.c: note: in included file (through include/linux/wait.h, include/linux/wait_bit.h, include/linux/fs.h):
./include/linux/list.h:229:25: warning: context imbalance in 'put_clnt_odstate' - unexpected unlock fs/nfsd/nfs4state.c:1188:9: warning: context imbalance in 'nfs4_put_stid' - unexpected unlock

I haven't looked at them closer. Could be you're well aware of them?

These warnings have been there forever. I'm told they are the result of bugs in sparse.

OK, good to know, thanks!

#8 lore raw

PATCH v4 2/4 NFSD: refactor nfsd_read_vector_dio to EVENT_CLASS useful for READ and WRITE

In reply to Mike Snitzer

Transform nfsd_read_vector_dio trace event into nfsd_analyze_dio_class and use it to create nfsd_analyze_read_dio and nfsd_analyze_write_dio trace events.

This prepares for nfsd_vfs_write() to also make use of it when handling misaligned WRITEs.

---
 fs/nfsd/trace.h | 52 ++++++++++++++++++++++++++++++++++++-------------
 fs/nfsd/vfs.c   | 11 ++++++-----
 2 files changed, 44 insertions(+), 19 deletions(-)
fs/nfsd/trace.h +38 -14
index 55055482f8a84..4173bd9344b6b 100644
@@ -473,25 +473,29 @@ DEFINE_NFSD_IO_EVENT(write_done);
 DEFINE_NFSD_IO_EVENT(commit_start);
 DEFINE_NFSD_IO_EVENT(commit_done);
 
-TRACE_EVENT(nfsd_read_vector_dio,
+DECLARE_EVENT_CLASS(nfsd_analyze_dio_class,
 	TP_PROTO(struct svc_rqst *rqstp,
 		 struct svc_fh	*fhp,
 		 u64		offset,
 		 u32		len,
-		 loff_t         start,
-		 loff_t         start_extra,
-		 loff_t         end,
-		 loff_t         end_extra),
-	TP_ARGS(rqstp, fhp, offset, len, start, start_extra, end, end_extra),
+		 loff_t		start,
+		 ssize_t	start_len,
+		 loff_t		middle,
+		 ssize_t	middle_len,
+		 loff_t		end,
+		 ssize_t	end_len),
+	TP_ARGS(rqstp, fhp, offset, len, start, start_len, middle, middle_len, end, end_len),
 	TP_STRUCT__entry(
 		__field(u32, xid)
 		__field(u32, fh_hash)
 		__field(u64, offset)
 		__field(u32, len)
 		__field(loff_t, start)
-		__field(loff_t, start_extra)
+		__field(ssize_t, start_len)
+		__field(loff_t, middle)
+		__field(ssize_t, middle_len)
 		__field(loff_t, end)
-		__field(loff_t, end_extra)
+		__field(ssize_t, end_len)
 	),
 	TP_fast_assign(
 		__entry->xid = be32_to_cpu(rqstp->rq_xid);
@@ -499,16 +503,36 @@ TRACE_EVENT(nfsd_read_vector_dio,
 		__entry->offset = offset;
 		__entry->len = len;
 		__entry->start = start;
-		__entry->start_extra = start_extra;
+		__entry->start_len = start_len;
+		__entry->middle = middle;
+		__entry->middle_len = middle_len;
 		__entry->end = end;
-		__entry->end_extra = end_extra;
+		__entry->end_len = end_len;
 	),
-	TP_printk("xid=0x%08x fh_hash=0x%08x offset=%llu len=%u start=%llu+%llu end=%llu-%llu",
+	TP_printk("xid=0x%08x fh_hash=0x%08x offset=%llu len=%u start=%llu+%lu middle=%llu+%lu end=%llu+%lu",
 		  __entry->xid, __entry->fh_hash,
 		  __entry->offset, __entry->len,
-		  __entry->start, __entry->start_extra,
-		  __entry->end, __entry->end_extra)
-);
+		  __entry->start, __entry->start_len,
+		  __entry->middle, __entry->middle_len,
+		  __entry->end, __entry->end_len)
+)
+
+#define DEFINE_NFSD_ANALYZE_DIO_EVENT(name)			\
+DEFINE_EVENT(nfsd_analyze_dio_class, nfsd_analyze_##name##_dio,	\
+	TP_PROTO(struct svc_rqst *rqstp,			\
+		 struct svc_fh	*fhp,				\
+		 u64		offset,				\
+		 u32		len,				\
+		 loff_t		start,				\
+		 ssize_t	start_len,			\
+		 loff_t		middle,				\
+		 ssize_t	middle_len,			\
+		 loff_t		end,				\
+		 ssize_t	end_len),			\
+	TP_ARGS(rqstp, fhp, offset, len, start, start_len, middle, middle_len, end, end_len))
+
+DEFINE_NFSD_ANALYZE_DIO_EVENT(read);
+DEFINE_NFSD_ANALYZE_DIO_EVENT(write);
 
 DECLARE_EVENT_CLASS(nfsd_err_class,
 	TP_PROTO(struct svc_rqst *rqstp,
fs/nfsd/vfs.c +6 -5
index e1751d3715264..609b85f8bde3e 100644
@@ -1094,7 +1094,7 @@ static bool nfsd_analyze_read_dio(struct svc_rqst *rqstp, struct svc_fh *fhp,
 				  struct nfsd_read_dio *read_dio)
 {
 	const u32 dio_blocksize = nf->nf_dio_read_offset_align;
-	loff_t orig_end = offset + len;
+	loff_t middle_end, orig_end = offset + len;
 
 	if (WARN_ONCE(!nf->nf_dio_mem_align || !nf->nf_dio_read_offset_align,
 		      "%s: underlying filesystem has not provided DIO alignment info\n",
@@ -1133,10 +1133,11 @@ static bool nfsd_analyze_read_dio(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	}
 
 	/* Show original offset and count, and how it was expanded for DIO */
-	trace_nfsd_read_vector_dio(rqstp, fhp, offset, len,
-				   read_dio->start, read_dio->start_extra,
-				   read_dio->end, read_dio->end_extra);
-
+	middle_end = read_dio->end - read_dio->end_extra;
+	trace_nfsd_analyze_read_dio(rqstp, fhp, offset, len,
+				    read_dio->start, read_dio->start_extra,
+				    offset, (middle_end - offset),
+				    middle_end, read_dio->end_extra);
 	return true;
 }
 
Signature · 1 line
2.44.0
#9 lore raw

PATCH v4 3/4 NFSD: prepare nfsd_vfs_write() to use O_DIRECT on misaligned WRITEs

In reply to Mike Snitzer

Refactor nfsd_vfs_write() to support splitting a WRITE into parts (which will be either misaligned or DIO-aligned). Doing so in a preliminary commit just allows for indentation and slight transformation to be more easily understood and reviewed.

---
 fs/nfsd/vfs.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
fs/nfsd/vfs.c +16 -9
index 609b85f8bde3e..0d4f9f452d466 100644
@@ -1342,7 +1342,6 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	struct super_block	*sb = file_inode(file)->i_sb;
 	struct kiocb		kiocb;
 	struct svc_export	*exp;
-	struct iov_iter		iter;
 	errseq_t		since;
 	__be32			nfserr;
 	int			host_err;
@@ -1350,6 +1349,9 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	unsigned int		pflags = current->flags;
 	bool			restore_flags = false;
 	unsigned int		nvecs;
+	struct iov_iter		iter_stack[1];
+	struct iov_iter		*iter = iter_stack;
+	unsigned int		n_iters = 0;
 
 	trace_nfsd_write_opened(rqstp, fhp, offset, *cnt);
 
@@ -1379,14 +1381,15 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		kiocb.ki_flags |= IOCB_DSYNC;
 
 	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
-	iov_iter_bvec(&iter, ITER_SOURCE, rqstp->rq_bvec, nvecs, *cnt);
+	iov_iter_bvec(&iter[0], ITER_SOURCE, rqstp->rq_bvec, nvecs, *cnt);
+	n_iters++;
 
 	switch (nfsd_io_cache_write) {
 	case NFSD_IO_DIRECT:
 		/* direct I/O must be aligned to device logical sector size */
 		if (nf->nf_dio_mem_align && nf->nf_dio_offset_align &&
 		    (((offset | *cnt) & (nf->nf_dio_offset_align-1)) == 0) &&
-		    iov_iter_is_aligned(&iter, nf->nf_dio_mem_align - 1,
+		    iov_iter_is_aligned(&iter[0], nf->nf_dio_mem_align - 1,
 					nf->nf_dio_offset_align - 1))
 			kiocb.ki_flags = IOCB_DIRECT;
 		break;
@@ -1400,13 +1403,17 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	since = READ_ONCE(file->f_wb_err);
 	if (verf)
 		nfsd_copy_write_verifier(verf, nn);
-	host_err = vfs_iocb_iter_write(file, &kiocb, &iter);
-	if (host_err < 0) {
-		commit_reset_write_verifier(nn, rqstp, host_err);
-		goto out_nfserr;
+	*cnt = 0;
+	for (int i = 0; i < n_iters; i++) {
+		host_err = vfs_iocb_iter_write(file, &kiocb, &iter[i]);
+		if (host_err < 0) {
+			commit_reset_write_verifier(nn, rqstp, host_err);
+			goto out_nfserr;
+		}
+		*cnt += host_err;
+		nfsd_stats_io_write_add(nn, exp, host_err);
 	}
-	*cnt = host_err;
-	nfsd_stats_io_write_add(nn, exp, *cnt);
+
 	fsnotify_modify(file);
 	host_err = filemap_check_wb_err(file->f_mapping, since);
 	if (host_err < 0)
Signature · 1 line
2.44.0
#10 lore raw

PATCH v4 4/4 NFSD: issue WRITEs using O_DIRECT even if IO is misaligned

In reply to Mike Snitzer

If NFSD_IO_DIRECT is used, split any misaligned WRITE into a start, middle and end as needed. The large middle extent is DIO-aligned and the start and/or end are misaligned. Buffered IO is used for the misaligned extents and O_DIRECT is used for the middle DIO-aligned extent.

The nfsd_analyze_write_dio trace event shows how NFSD splits a given misaligned WRITE into a mix of misaligned extent(s) and a DIO-aligned extent.

This combination of trace events is useful:

echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_analyze_write_dio/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable

Which for this dd command:

dd if=/dev/zero of=/mnt/share1/test bs=47008 count=2 oflag=direct

Results in:

  nfsd-55714   [043] ..... 79976.260851: nfsd_write_opened: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.260852: nfsd_analyze_write_dio: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008 start=0+0 middle=0+45056 end=45056+1952
  nfsd-55714   [043] ..... 79976.260857: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0x0 pos 0x0 bytecount 0xb000
  nfsd-55714   [043] ..... 79976.260965: nfsd_write_io_done: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_write_opened: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_analyze_write_dio: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008 start=47008+2144 middle=49152+40960 end=90112+3904
  nfsd-55714   [043] ..... 79976.307797: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0xc000 pos 0xc000 bytecount 0xa000
  nfsd-55714   [043] ..... 79976.307866: nfsd_write_io_done: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
---
 fs/nfsd/vfs.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 131 insertions(+), 11 deletions(-)
fs/nfsd/vfs.c +131 -11
index 0d4f9f452d466..4980800fab66e 100644
@@ -1315,6 +1315,121 @@ static int wait_for_concurrent_writes(struct file *file)
 	return err;
 }
 
+struct nfsd_write_dio
+{
+	loff_t middle_offset;	/* Offset for start of DIO-aligned middle */
+	loff_t end_offset;	/* Offset for start of DIO-aligned end */
+	ssize_t	start_len;	/* Length for misaligned first extent */
+	ssize_t	middle_len;	/* Length for DIO-aligned middle extent */
+	ssize_t	end_len;	/* Length for misaligned last extent */
+};
+
+static void init_nfsd_write_dio(struct nfsd_write_dio *write_dio)
+{
+	memset(write_dio, 0, sizeof(*write_dio));
+}
+
+static bool nfsd_analyze_write_dio(struct svc_rqst *rqstp, struct svc_fh *fhp,
+				   struct nfsd_file *nf, loff_t offset,
+				   unsigned long len, struct nfsd_write_dio *write_dio)
+{
+	const u32 dio_blocksize = nf->nf_dio_offset_align;
+	loff_t orig_end, middle_end, start_end, start_offset = offset;
+	ssize_t start_len = len;
+	bool aligned = true;
+
+	if (WARN_ONCE(!nf->nf_dio_mem_align || !dio_blocksize,
+		      "%s: underlying filesystem has not provided DIO alignment info\n",
+		      __func__))
+		return false;
+
+	if (WARN_ONCE(dio_blocksize > PAGE_SIZE,
+		      "%s: underlying storage's dio_blocksize=%u > PAGE_SIZE=%lu\n",
+		      __func__, dio_blocksize, PAGE_SIZE))
+		return false;
+
+	if (unlikely(len < dio_blocksize)) {
+		aligned = false;
+		goto out;
+	}
+
+	if (((offset | len) & (dio_blocksize-1)) == 0) {
+		/* already DIO-aligned, no misaligned head or tail */
+		write_dio->middle_offset = offset;
+		write_dio->middle_len = len;
+		/* clear these for the benefit of trace_nfsd_analyze_write_dio */
+		start_offset = 0;
+		start_len = 0;
+		goto out;
+	}
+
+	start_end = round_up(offset, dio_blocksize);
+	start_len = start_end - offset;
+	orig_end = offset + len;
+	middle_end = round_down(orig_end, dio_blocksize);
+
+	write_dio->start_len = start_len;
+	write_dio->middle_offset = start_end;
+	write_dio->middle_len = middle_end - start_end;
+	write_dio->end_offset = middle_end;
+	write_dio->end_len = orig_end - middle_end;
+out:
+	trace_nfsd_analyze_write_dio(rqstp, fhp, offset, len, start_offset, start_len,
+				     write_dio->middle_offset, write_dio->middle_len,
+				     write_dio->end_offset, write_dio->end_len);
+	return aligned;
+}
+
+/*
+ * Setup as many as 3 iov_iter based on extents possibly described by @write_dio.
+ * @iterp: pointer to pointer to onstack array of 3 iov_iter structs from caller.
+ * @iter_is_dio_aligned: pointer to onstack array of 3 bools from caller.
+ * @dio_aligned: bool that reflects nfsd_analyze_write_dio()'s return
+ * @rq_bvec: backing bio_vec used to setup all 3 iov_iter permutations.
+ * @nvecs: number of segments in @rq_bvec
+ * @cnt: size of the request in bytes
+ * @write_dio: nfsd_write_dio struct that describes start, middle and end extents.
+ *
+ * Returns the number of iov_iter that were setup.
+ */
+static int nfsd_setup_write_iters(struct iov_iter **iterp, bool *iter_is_dio_aligned,
+				  bool dio_aligned, struct bio_vec *rq_bvec,
+				  unsigned int nvecs, unsigned long cnt,
+				  struct nfsd_write_dio *write_dio)
+{
+	int n_iters = 0;
+	struct iov_iter *iters = *iterp;
+
+	/* Setup misaligned start? */
+	if (write_dio->start_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iters[n_iters].count = write_dio->start_len;
+		n_iters++;
+	}
+
+	/* Setup possibly DIO-aligned middle */
+	iter_is_dio_aligned[n_iters] = dio_aligned;
+	iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+	if (dio_aligned) {
+		if (write_dio->start_len)
+			iov_iter_advance(&iters[n_iters], write_dio->start_len);
+		iters[n_iters].count -= write_dio->end_len;
+	}
+	n_iters++;
+
+	/* Setup misaligned end? */
+	if (write_dio->end_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iov_iter_advance(&iters[n_iters],
+				 write_dio->start_len + write_dio->middle_len);
+		n_iters++;
+	}
+
+	return n_iters;
+}
+
 /**
  * nfsd_vfs_write - write data to an already-open file
  * @rqstp: RPC execution context
@@ -1349,9 +1464,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	unsigned int		pflags = current->flags;
 	bool			restore_flags = false;
 	unsigned int		nvecs;
-	struct iov_iter		iter_stack[1];
+	struct iov_iter		iter_stack[3];
 	struct iov_iter		*iter = iter_stack;
 	unsigned int		n_iters = 0;
+	bool			iov_iter_is_dio_aligned[3];
+	bool			dio_aligned = false;
+	struct nfsd_write_dio	write_dio;
 
 	trace_nfsd_write_opened(rqstp, fhp, offset, *cnt);
 
@@ -1380,18 +1498,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (stable && !fhp->fh_use_wgather)
 		kiocb.ki_flags |= IOCB_DSYNC;
 
-	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
-	iov_iter_bvec(&iter[0], ITER_SOURCE, rqstp->rq_bvec, nvecs, *cnt);
-	n_iters++;
-
+	init_nfsd_write_dio(&write_dio);
 	switch (nfsd_io_cache_write) {
 	case NFSD_IO_DIRECT:
-		/* direct I/O must be aligned to device logical sector size */
-		if (nf->nf_dio_mem_align && nf->nf_dio_offset_align &&
-		    (((offset | *cnt) & (nf->nf_dio_offset_align-1)) == 0) &&
-		    iov_iter_is_aligned(&iter[0], nf->nf_dio_mem_align - 1,
-					nf->nf_dio_offset_align - 1))
-			kiocb.ki_flags = IOCB_DIRECT;
+		if (nfsd_analyze_write_dio(rqstp, fhp, nf, offset,
+					   *cnt, &write_dio))
+			dio_aligned = true;
 		break;
 	case NFSD_IO_DONTCACHE:
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1400,11 +1512,19 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		break;
 	}
 
+	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
+	n_iters = nfsd_setup_write_iters(&iter, iov_iter_is_dio_aligned, dio_aligned,
+					 rqstp->rq_bvec, nvecs, *cnt, &write_dio);
+
 	since = READ_ONCE(file->f_wb_err);
 	if (verf)
 		nfsd_copy_write_verifier(verf, nn);
 	*cnt = 0;
 	for (int i = 0; i < n_iters; i++) {
+		if (iov_iter_is_dio_aligned[i])
+			kiocb.ki_flags |= IOCB_DIRECT;
+		else
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		host_err = vfs_iocb_iter_write(file, &kiocb, &iter[i]);
 		if (host_err < 0) {
 			commit_reset_write_verifier(nn, rqstp, host_err);
Signature · 1 line
2.44.0
#11 lore raw

Re: [PATCH v4 4/4] NFSD: issue WRITEs using O_DIRECT even if IO is misaligned

Quoted 44 lines · Mike Snitzer

If NFSD_IO_DIRECT is used, split any misaligned WRITE into a start, middle and end as needed. The large middle extent is DIO-aligned and the start and/or end are misaligned. Buffered IO is used for the misaligned extents and O_DIRECT is used for the middle DIO-aligned extent.

The nfsd_analyze_write_dio trace event shows how NFSD splits a given misaligned WRITE into a mix of misaligned extent(s) and a DIO-aligned extent.

This combination of trace events is useful:

echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_analyze_write_dio/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable

Which for this dd command:

dd if=/dev/zero of=/mnt/share1/test bs=47008 count=2 oflag=direct

Results in:

  nfsd-55714   [043] ..... 79976.260851: nfsd_write_opened: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.260852: nfsd_analyze_write_dio: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008 start=0+0 middle=0+45056 end=45056+1952
  nfsd-55714   [043] ..... 79976.260857: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0x0 pos 0x0 bytecount 0xb000
  nfsd-55714   [043] ..... 79976.260965: nfsd_write_io_done: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_write_opened: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_analyze_write_dio: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008 start=47008+2144 middle=49152+40960 end=90112+3904
  nfsd-55714   [043] ..... 79976.307797: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0xc000 pos 0xc000 bytecount 0xa000
  nfsd-55714   [043] ..... 79976.307866: nfsd_write_io_done: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
---
 fs/nfsd/vfs.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 131 insertions(+), 11 deletions(-)
fs/nfsd/vfs.c +0 -0
index 0d4f9f452d466..4980800fab66e 100644

fs/nfsd/vfs.c static int wait_for_concurrent_writes(struct file *file) · line 1319

Quoted 5 lines
fs/nfsd/vfs.c +2 -0
@@ -1315,6 +1315,121 @@ static int wait_for_concurrent_writes(struct file *file)
 	return err;
 }
 
+struct nfsd_write_dio
+{

struct nfsd_write_dio {

Quoted 117 lines
fs/nfsd/vfs.c +113 -0
+	loff_t middle_offset;	/* Offset for start of DIO-aligned middle */
+	loff_t end_offset;	/* Offset for start of DIO-aligned end */
+	ssize_t	start_len;	/* Length for misaligned first extent */
+	ssize_t	middle_len;	/* Length for DIO-aligned middle extent */
+	ssize_t	end_len;	/* Length for misaligned last extent */
+};
+
+static void init_nfsd_write_dio(struct nfsd_write_dio *write_dio)
+{
+	memset(write_dio, 0, sizeof(*write_dio));
+}
+
+static bool nfsd_analyze_write_dio(struct svc_rqst *rqstp, struct svc_fh *fhp,
+				   struct nfsd_file *nf, loff_t offset,
+				   unsigned long len, struct nfsd_write_dio *write_dio)
+{
+	const u32 dio_blocksize = nf->nf_dio_offset_align;
+	loff_t orig_end, middle_end, start_end, start_offset = offset;
+	ssize_t start_len = len;
+	bool aligned = true;
+
+	if (WARN_ONCE(!nf->nf_dio_mem_align || !dio_blocksize,
+		      "%s: underlying filesystem has not provided DIO alignment info\n",
+		      __func__))
+		return false;
+
+	if (WARN_ONCE(dio_blocksize > PAGE_SIZE,
+		      "%s: underlying storage's dio_blocksize=%u > PAGE_SIZE=%lu\n",
+		      __func__, dio_blocksize, PAGE_SIZE))
+		return false;
+
+	if (unlikely(len < dio_blocksize)) {
+		aligned = false;
+		goto out;
+	}
+
+	if (((offset | len) & (dio_blocksize-1)) == 0) {
+		/* already DIO-aligned, no misaligned head or tail */
+		write_dio->middle_offset = offset;
+		write_dio->middle_len = len;
+		/* clear these for the benefit of trace_nfsd_analyze_write_dio */
+		start_offset = 0;
+		start_len = 0;
+		goto out;
+	}
+
+	start_end = round_up(offset, dio_blocksize);
+	start_len = start_end - offset;
+	orig_end = offset + len;
+	middle_end = round_down(orig_end, dio_blocksize);
+
+	write_dio->start_len = start_len;
+	write_dio->middle_offset = start_end;
+	write_dio->middle_len = middle_end - start_end;
+	write_dio->end_offset = middle_end;
+	write_dio->end_len = orig_end - middle_end;
+out:
+	trace_nfsd_analyze_write_dio(rqstp, fhp, offset, len, start_offset, start_len,
+				     write_dio->middle_offset, write_dio->middle_len,
+				     write_dio->end_offset, write_dio->end_len);
+	return aligned;
+}
+
+/*
+ * Setup as many as 3 iov_iter based on extents possibly described by @write_dio.
+ * @iterp: pointer to pointer to onstack array of 3 iov_iter structs from caller.
+ * @iter_is_dio_aligned: pointer to onstack array of 3 bools from caller.
+ * @dio_aligned: bool that reflects nfsd_analyze_write_dio()'s return
+ * @rq_bvec: backing bio_vec used to setup all 3 iov_iter permutations.
+ * @nvecs: number of segments in @rq_bvec
+ * @cnt: size of the request in bytes
+ * @write_dio: nfsd_write_dio struct that describes start, middle and end extents.
+ *
+ * Returns the number of iov_iter that were setup.
+ */
+static int nfsd_setup_write_iters(struct iov_iter **iterp, bool *iter_is_dio_aligned,
+				  bool dio_aligned, struct bio_vec *rq_bvec,
+				  unsigned int nvecs, unsigned long cnt,
+				  struct nfsd_write_dio *write_dio)
+{
+	int n_iters = 0;
+	struct iov_iter *iters = *iterp;
+
+	/* Setup misaligned start? */
+	if (write_dio->start_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iters[n_iters].count = write_dio->start_len;
+		n_iters++;
+	}
+
+	/* Setup possibly DIO-aligned middle */
+	iter_is_dio_aligned[n_iters] = dio_aligned;
+	iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+	if (dio_aligned) {
+		if (write_dio->start_len)
+			iov_iter_advance(&iters[n_iters], write_dio->start_len);
+		iters[n_iters].count -= write_dio->end_len;
+	}
+	n_iters++;
+
+	/* Setup misaligned end? */
+	if (write_dio->end_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iov_iter_advance(&iters[n_iters],
+				 write_dio->start_len + write_dio->middle_len);
+		n_iters++;
+	}
+
+	return n_iters;
+}
+
 /**
  * nfsd_vfs_write - write data to an already-open file
  * @rqstp: RPC execution context

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1467

Quoted 5 lines
fs/nfsd/vfs.c +1 -1
@@ -1349,9 +1464,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	unsigned int		pflags = current->flags;
 	bool			restore_flags = false;
 	unsigned int		nvecs;
-	struct iov_iter		iter_stack[1];
+	struct iov_iter		iter_stack[3];

struct iov_iter isn't that small. This is going to grow the stack frame substantially but is used for only the direct I/O case.

Quoted 9 lines
fs/nfsd/vfs.c +3 -0
 	struct iov_iter		*iter = iter_stack;
 	unsigned int		n_iters = 0;
+	bool			iov_iter_is_dio_aligned[3];
+	bool			dio_aligned = false;
+	struct nfsd_write_dio	write_dio;
 
 	trace_nfsd_write_opened(rqstp, fhp, offset, *cnt);
 

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1501

Quoted 8 lines
fs/nfsd/vfs.c +1 -4
@@ -1380,18 +1498,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (stable && !fhp->fh_use_wgather)
 		kiocb.ki_flags |= IOCB_DSYNC;
 
-	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
-	iov_iter_bvec(&iter[0], ITER_SOURCE, rqstp->rq_bvec, nvecs, *cnt);
-	n_iters++;
-
+	init_nfsd_write_dio(&write_dio);

I assume init_nfsd_write_dio() is going to be called only once. Is there a plan to make it more than a memset() ? Can it be called in only direct I/O mode?

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1506

Quoted 11 lines
fs/nfsd/vfs.c +3 -6
 	switch (nfsd_io_cache_write) {
 	case NFSD_IO_DIRECT:
-		/* direct I/O must be aligned to device logical sector size */
-		if (nf->nf_dio_mem_align && nf->nf_dio_offset_align &&
-		    (((offset | *cnt) & (nf->nf_dio_offset_align-1)) == 0) &&
-		    iov_iter_is_aligned(&iter[0], nf->nf_dio_mem_align - 1,
-					nf->nf_dio_offset_align - 1))
-			kiocb.ki_flags = IOCB_DIRECT;
+		if (nfsd_analyze_write_dio(rqstp, fhp, nf, offset,
+					   *cnt, &write_dio))
+			dio_aligned = true;

How about

		dio_aligned = nfsd_analyze_write_dio(rqstp, fhp, nf,
						     offset, *cnt,
						     &write_dio);

Let's make nfsd_analyze_write_dio a "noinline" so that the compiler removes it from the hot path in page cache I/O mode.

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1517

Quoted 10 lines
fs/nfsd/vfs.c +3 -0
 		break;
 	case NFSD_IO_DONTCACHE:
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1400,11 +1512,19 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		break;
 	}
 
+	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
+	n_iters = nfsd_setup_write_iters(&iter, iov_iter_is_dio_aligned, dio_aligned,
+					 rqstp->rq_bvec, nvecs, *cnt, &write_dio);

Is there a plan to use buffer re-alignment for the other two I/O modes?

I ask because there are many more conditional branches now, and they seem to be useful only if there are multiple iters. And it looks like there are multiple iters only in the direct I/O case.

Generally what we do in situations like this is create utility functions that contain code common to all paths, and have the separate paths use those helpers in the combination that they need. Not only is the instruction path length shorter for each individual path, but the resulting source code is much more legible.

Quoted 13 lines
fs/nfsd/vfs.c +5 -0
+
 	since = READ_ONCE(file->f_wb_err);
 	if (verf)
 		nfsd_copy_write_verifier(verf, nn);
 	*cnt = 0;
 	for (int i = 0; i < n_iters; i++) {
+		if (iov_iter_is_dio_aligned[i])
+			kiocb.ki_flags |= IOCB_DIRECT;
+		else
+			kiocb.ki_flags &= ~IOCB_DIRECT;
 		host_err = vfs_iocb_iter_write(file, &kiocb, &iter[i]);
 		if (host_err < 0) {
 			commit_reset_write_verifier(nn, rqstp, host_err);
Signature · 1 line
Chuck Lever
#12 lore raw

Re: [PATCH v4 4/4] NFSD: issue WRITEs using O_DIRECT even if IO is misaligned

Quoted 48 lines · Chuck Lever
Quoted 47 lines · Mike Snitzer

If NFSD_IO_DIRECT is used, split any misaligned WRITE into a start, middle and end as needed. The large middle extent is DIO-aligned and the start and/or end are misaligned. Buffered IO is used for the misaligned extents and O_DIRECT is used for the middle DIO-aligned extent.

The nfsd_analyze_write_dio trace event shows how NFSD splits a given misaligned WRITE into a mix of misaligned extent(s) and a DIO-aligned extent.

This combination of trace events is useful:

echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_analyze_write_dio/enable echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable

Which for this dd command:

dd if=/dev/zero of=/mnt/share1/test bs=47008 count=2 oflag=direct

Results in:

  nfsd-55714   [043] ..... 79976.260851: nfsd_write_opened: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.260852: nfsd_analyze_write_dio: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008 start=0+0 middle=0+45056 end=45056+1952
  nfsd-55714   [043] ..... 79976.260857: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0x0 pos 0x0 bytecount 0xb000
  nfsd-55714   [043] ..... 79976.260965: nfsd_write_io_done: xid=0x966c5d2d fh_hash=0x4d34e6c1 offset=0 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_write_opened: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
  nfsd-55714   [043] ..... 79976.307762: nfsd_analyze_write_dio: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008 start=47008+2144 middle=49152+40960 end=90112+3904
  nfsd-55714   [043] ..... 79976.307797: xfs_file_direct_write: dev 259:12 ino 0x3e00008f disize 0xc000 pos 0xc000 bytecount 0xa000
  nfsd-55714   [043] ..... 79976.307866: nfsd_write_io_done: xid=0x67e5ce6f fh_hash=0x4d34e6c1 offset=47008 len=47008
---
 fs/nfsd/vfs.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 131 insertions(+), 11 deletions(-)
fs/nfsd/vfs.c +0 -0
index 0d4f9f452d466..4980800fab66e 100644
@@ -1315,6 +1315,121 @@ static int wait_for_concurrent_writes(struct file *file)
 	return err;
 }
 
Quoted 4 lines

fs/nfsd/vfs.c static int wait_for_concurrent_writes(struct file *file) · line 1319

Quoted 2 lines
fs/nfsd/vfs.c +2 -0
+struct nfsd_write_dio
+{

struct nfsd_write_dio {

Yeap, fixed now ;)

Quoted 117 lines
Quoted 117 lines
fs/nfsd/vfs.c +113 -0
+	loff_t middle_offset;	/* Offset for start of DIO-aligned middle */
+	loff_t end_offset;	/* Offset for start of DIO-aligned end */
+	ssize_t	start_len;	/* Length for misaligned first extent */
+	ssize_t	middle_len;	/* Length for DIO-aligned middle extent */
+	ssize_t	end_len;	/* Length for misaligned last extent */
+};
+
+static void init_nfsd_write_dio(struct nfsd_write_dio *write_dio)
+{
+	memset(write_dio, 0, sizeof(*write_dio));
+}
+
+static bool nfsd_analyze_write_dio(struct svc_rqst *rqstp, struct svc_fh *fhp,
+				   struct nfsd_file *nf, loff_t offset,
+				   unsigned long len, struct nfsd_write_dio *write_dio)
+{
+	const u32 dio_blocksize = nf->nf_dio_offset_align;
+	loff_t orig_end, middle_end, start_end, start_offset = offset;
+	ssize_t start_len = len;
+	bool aligned = true;
+
+	if (WARN_ONCE(!nf->nf_dio_mem_align || !dio_blocksize,
+		      "%s: underlying filesystem has not provided DIO alignment info\n",
+		      __func__))
+		return false;
+
+	if (WARN_ONCE(dio_blocksize > PAGE_SIZE,
+		      "%s: underlying storage's dio_blocksize=%u > PAGE_SIZE=%lu\n",
+		      __func__, dio_blocksize, PAGE_SIZE))
+		return false;
+
+	if (unlikely(len < dio_blocksize)) {
+		aligned = false;
+		goto out;
+	}
+
+	if (((offset | len) & (dio_blocksize-1)) == 0) {
+		/* already DIO-aligned, no misaligned head or tail */
+		write_dio->middle_offset = offset;
+		write_dio->middle_len = len;
+		/* clear these for the benefit of trace_nfsd_analyze_write_dio */
+		start_offset = 0;
+		start_len = 0;
+		goto out;
+	}
+
+	start_end = round_up(offset, dio_blocksize);
+	start_len = start_end - offset;
+	orig_end = offset + len;
+	middle_end = round_down(orig_end, dio_blocksize);
+
+	write_dio->start_len = start_len;
+	write_dio->middle_offset = start_end;
+	write_dio->middle_len = middle_end - start_end;
+	write_dio->end_offset = middle_end;
+	write_dio->end_len = orig_end - middle_end;
+out:
+	trace_nfsd_analyze_write_dio(rqstp, fhp, offset, len, start_offset, start_len,
+				     write_dio->middle_offset, write_dio->middle_len,
+				     write_dio->end_offset, write_dio->end_len);
+	return aligned;
+}
+
+/*
+ * Setup as many as 3 iov_iter based on extents possibly described by @write_dio.
+ * @iterp: pointer to pointer to onstack array of 3 iov_iter structs from caller.
+ * @iter_is_dio_aligned: pointer to onstack array of 3 bools from caller.
+ * @dio_aligned: bool that reflects nfsd_analyze_write_dio()'s return
+ * @rq_bvec: backing bio_vec used to setup all 3 iov_iter permutations.
+ * @nvecs: number of segments in @rq_bvec
+ * @cnt: size of the request in bytes
+ * @write_dio: nfsd_write_dio struct that describes start, middle and end extents.
+ *
+ * Returns the number of iov_iter that were setup.
+ */
+static int nfsd_setup_write_iters(struct iov_iter **iterp, bool *iter_is_dio_aligned,
+				  bool dio_aligned, struct bio_vec *rq_bvec,
+				  unsigned int nvecs, unsigned long cnt,
+				  struct nfsd_write_dio *write_dio)
+{
+	int n_iters = 0;
+	struct iov_iter *iters = *iterp;
+
+	/* Setup misaligned start? */
+	if (write_dio->start_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iters[n_iters].count = write_dio->start_len;
+		n_iters++;
+	}
+
+	/* Setup possibly DIO-aligned middle */
+	iter_is_dio_aligned[n_iters] = dio_aligned;
+	iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+	if (dio_aligned) {
+		if (write_dio->start_len)
+			iov_iter_advance(&iters[n_iters], write_dio->start_len);
+		iters[n_iters].count -= write_dio->end_len;
+	}
+	n_iters++;
+
+	/* Setup misaligned end? */
+	if (write_dio->end_len) {
+		iter_is_dio_aligned[n_iters] = false;
+		iov_iter_bvec(&iters[n_iters], ITER_SOURCE, rq_bvec, nvecs, cnt);
+		iov_iter_advance(&iters[n_iters],
+				 write_dio->start_len + write_dio->middle_len);
+		n_iters++;
+	}
+
+	return n_iters;
+}
+
 /**
  * nfsd_vfs_write - write data to an already-open file
  * @rqstp: RPC execution context
@@ -1349,9 +1464,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
Quoted 8 lines

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1467

Quoted 5 lines
fs/nfsd/vfs.c +1 -1
 	unsigned int		pflags = current->flags;
 	bool			restore_flags = false;
 	unsigned int		nvecs;
-	struct iov_iter		iter_stack[1];
+	struct iov_iter		iter_stack[3];

struct iov_iter isn't that small. This is going to grow the stack frame substantially but is used for only the direct I/O case.

Yes, that's the only lingering footprint I have after another pass at cleanup based on your IPL feedback yesterday.

I expect to be able to push the use of multiple iov_iter down into the O_DIRECT path only.

Quoted 12 lines
Quoted 12 lines
fs/nfsd/vfs.c +3 -0
 	struct iov_iter		*iter = iter_stack;
 	unsigned int		n_iters = 0;
+	bool			iov_iter_is_dio_aligned[3];
+	bool			dio_aligned = false;
+	struct nfsd_write_dio	write_dio;
 
 	trace_nfsd_write_opened(rqstp, fhp, offset, *cnt);
 
@@ -1380,18 +1498,12 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (stable && !fhp->fh_use_wgather)
 		kiocb.ki_flags |= IOCB_DSYNC;
 
Quoted 9 lines

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1501

Quoted 5 lines
fs/nfsd/vfs.c +1 -4
-	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
-	iov_iter_bvec(&iter[0], ITER_SOURCE, rqstp->rq_bvec, nvecs, *cnt);
-	n_iters++;
-
+	init_nfsd_write_dio(&write_dio);

I assume init_nfsd_write_dio() is going to be called only once. Is there a plan to make it more than a memset() ? Can it be called in only direct I/O mode?

Yes, I've fixed this.

Quoted 14 lines

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1506

Quoted 11 lines
fs/nfsd/vfs.c +3 -6
 	switch (nfsd_io_cache_write) {
 	case NFSD_IO_DIRECT:
-		/* direct I/O must be aligned to device logical sector size */
-		if (nf->nf_dio_mem_align && nf->nf_dio_offset_align &&
-		    (((offset | *cnt) & (nf->nf_dio_offset_align-1)) == 0) &&
-		    iov_iter_is_aligned(&iter[0], nf->nf_dio_mem_align - 1,
-					nf->nf_dio_offset_align - 1))
-			kiocb.ki_flags = IOCB_DIRECT;
+		if (nfsd_analyze_write_dio(rqstp, fhp, nf, offset,
+					   *cnt, &write_dio))
+			dio_aligned = true;

How about

Quoted 3 lines
		dio_aligned = nfsd_analyze_write_dio(rqstp, fhp, nf,
						     offset, *cnt,
						     &write_dio);

I've iterated on things a bit, no longer need dio_aligned variable.

Quoted 2 lines

Let's make nfsd_analyze_write_dio a "noinline" so that the compiler removes it from the hot path in page cache I/O mode.

OK, will do.

Quoted 17 lines

fs/nfsd/vfs.c nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, · line 1517

Quoted 10 lines
fs/nfsd/vfs.c +3 -0
 		break;
 	case NFSD_IO_DONTCACHE:
 		kiocb.ki_flags = IOCB_DONTCACHE;
@@ -1400,11 +1512,19 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		break;
 	}
 
+	nvecs = xdr_buf_to_bvec(rqstp->rq_bvec, rqstp->rq_maxpages, payload);
+	n_iters = nfsd_setup_write_iters(&iter, iov_iter_is_dio_aligned, dio_aligned,
+					 rqstp->rq_bvec, nvecs, *cnt, &write_dio);

Is there a plan to use buffer re-alignment for the other two I/O modes?

I ask because there are many more conditional branches now, and they seem to be useful only if there are multiple iters. And it looks like there are multiple iters only in the direct I/O case.

Quoted 5 lines

Generally what we do in situations like this is create utility functions that contain code common to all paths, and have the separate paths use those helpers in the combination that they need. Not only is the instruction path length shorter for each individual path, but the resulting source code is much more legible.

Yes, I am now working on doing just that.

Thanks,
Mike