Skip to contentSkip to conversation

NFS/localio: issue IO inline when not reclaiming memory

PATCH New Mike Snitzer 1 participant 4 messages Jul 6

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover NFS/localio: issue IO inline when not reclaiming memory

Hi,

Every NFS LOCALIO read, write and commit is currently issued indirectly: it is queued onto the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than being submitted to the underlying filesystem in the calling context. That intermediate hop was introduced by commit b9f5dd57f4a5 ("nfs/localio:
use dedicated workqueues for filesystem read and write") because:
LOCALIO submits IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing so from a WQ_MEM_RECLAIM worker -- most importantly writeback's wb_workfn running on bdi_wq -- or from an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock.

However, that hazard only exists when the submitting context is itself a memory-reclaim context. For ordinary application/task submission -- e.g. O_DIRECT, or an fsync-driven commit -- the workqueue hop buys nothing: it just adds a context switch and scheduling latency per IO and throws away the NFS client's inherent application-context parallelism.

This series makes the hop conditional. It adds a small workqueue-core helper, current_is_workqueue_mem_reclaim(), that reports whether %current is a WQ_MEM_RECLAIM worker using exactly the predicate check_flush_dependency() warns on. LOCALIO uses it (together with the existing PF_MEMALLOC test) in a new nfs_local_defer_io() helper to decide per-IO whether it must defer to nfslocaliod_workqueue or may issue the IO inline. Reclaim contexts still defer and are unaffected; everything else runs inline.

Patch 1 adds current_is_workqueue_mem_reclaim() and applies the gating to
the read and write paths.
Patch 2 removes never-taken FLUSH_SYNC handling from nfs_local_commit()
(every caller supplies a FLUSH_SYNC-stripped "how"), which also drops the sole user of the ctx->done completion plumbing and the now-unused "how" argument. No functional change.
Patch 3 extends the same gating to the commit (fsync) path. Note the
writeback-triggered commit does reach here in reclaim context -- nfs_write_inode() (the ->write_inode super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq -- so that case correctly keeps deferring; only app-context commits run inline.

Patch 1 touches the workqueue core (kernel/workqueue.c, include/linux/ workqueue.h), hence the Cc to Tejun Heo, Lai Jiangshan and LKML. The new export mirrors the existing current_is_workqueue_rescuer()/current_work() context-introspection helpers.

Additional note for reviewers:
- The inline path now stacks the NFS pgio path plus the underlying filesystem's ->write_iter/->read_iter (and vfs_fsync_range) in task context. The reclaim-context stack-depth concern of b9f5dd57f4a5 (2) is avoided for the deferred paths; feedback on inline stack headroom under deeply-stacking filesystems (XFS especially) is welcome.

All review appreciated, thanks.
Mike

Mike Snitzer (3):
NFS/localio: issue IO inline when not in a memory-reclaim context NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit NFS/localio: issue commit inline when not in a memory-reclaim context

 fs/nfs/internal.h         |  4 +--
 fs/nfs/localio.c          | 56 ++++++++++++++++++++++++++++++---------
 fs/nfs/write.c            |  2 +-
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 24 +++++++++++++++++
 5 files changed, 71 insertions(+), 16 deletions(-)
Signature · 1 line
2.44.0
#1 lore raw

PATCH 1/3 NFS/localio: issue IO inline when not in a memory-reclaim context

Every LOCALIO read and write is currently bounced through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually required when the submitting context is a memory-reclaim context: LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock, which is why commit b9f5dd57f4a5 ("nfs/localio: use dedicated workqueues for filesystem read and write") introduced the intermediate workqueue.

Outside of reclaim context -- ordinary application/task submission such as O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and merely adds a context switch and scheduling latency per IO while discarding the NFS client's inherent application-context parallelism.

Add current_is_workqueue_mem_reclaim(), which reports whether %current is a WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency() warns on. Use it, together with the PF_MEMALLOC check, in the new nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read() and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue the IO inline. Buffered writeback continues to bounce (wb_workfn is a WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.

Running nfs_local_call_write() inline is safe: it already saves and restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it sets and scopes the file opener's creds. The async O_DIRECT completion path is likewise unaffected: when the underlying filesystem returns -EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete / nfs_local_write_aio_complete) can run in bottom-half context and so must still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is independent of how the IO was submitted, and this change leaves it as-is; only the submission side stops unconditionally hopping through nfslocaliod_workqueue.

Assisted-by Claude:claude-opus-4-8
---
 fs/nfs/localio.c          | 33 +++++++++++++++++++++++++++++++--
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 24 ++++++++++++++++++++++++
 3 files changed, 56 insertions(+), 2 deletions(-)
fs/nfs/localio.c +31 -2
index e55c5977fcc3..d3e480888eb1 100644
@@ -699,6 +699,29 @@ static void nfs_local_call_read(struct work_struct *work)
 	}
 }
 
+/*
+ * Decide whether LOCALIO must defer submission to the dedicated
+ * !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than issue the IO inline.
+ *
+ * LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS),
+ * which may in turn flush its own !WQ_MEM_RECLAIM workqueue.  Doing so from a
+ * memory-reclaim context -- either a WQ_MEM_RECLAIM worker (most importantly
+ * writeback's wb_workfn running on bdi_wq) or an explicit reclaim task
+ * (PF_MEMALLOC) -- would trip check_flush_dependency() and risks a
+ * forward-progress deadlock; see commit b9f5dd57f4a5 ("nfs/localio: use
+ * dedicated workqueues for filesystem read and write").  In that case defer
+ * to nfslocaliod_workqueue.
+ *
+ * Otherwise (ordinary application/task context, e.g. O_DIRECT or fsync-driven
+ * submission) issue the IO inline: this preserves the NFS client's inherent
+ * application-context parallelism and avoids the per-IO workqueue hop.
+ */
+static inline bool nfs_local_defer_io(void)
+{
+	return (current->flags & PF_MEMALLOC) ||
+		current_is_workqueue_mem_reclaim();
+}
+
 static void nfs_local_do_read(struct nfs_local_kiocb *iocb,
 			      const struct rpc_call_ops *call_ops)
 {
@@ -711,7 +734,10 @@ static void nfs_local_do_read(struct nfs_local_kiocb *iocb,
 	hdr->res.eof = false;
 
 	INIT_WORK(&iocb->work, nfs_local_call_read);
-	queue_work(nfslocaliod_workqueue, &iocb->work);
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &iocb->work);
+	else
+		nfs_local_call_read(&iocb->work);
 }
 
 static void
@@ -929,7 +955,10 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb,
 	nfs_set_local_verifier(hdr->inode, hdr->res.verf, hdr->args.stable);
 
 	INIT_WORK(&iocb->work, nfs_local_call_write);
-	queue_work(nfslocaliod_workqueue, &iocb->work);
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &iocb->work);
+	else
+		nfs_local_call_write(&iocb->work);
 }
 
 static struct nfs_local_kiocb *
include/linux/workqueue.h +1 -0
index bc1ccdfbfb1d..3d2e426bcf27 100644
@@ -659,6 +659,7 @@ extern void workqueue_set_min_active(struct workqueue_struct *wq,
 				     int min_active);
 extern struct work_struct *current_work(void);
 extern bool current_is_workqueue_rescuer(void);
+extern bool current_is_workqueue_mem_reclaim(void);
 extern bool workqueue_congested(int cpu, struct workqueue_struct *wq);
 extern unsigned int work_busy(struct work_struct *work);
 extern __printf(1, 2) void set_worker_desc(const char *fmt, ...);
kernel/workqueue.c +24 -0
index 03d9588e16d7..4add75c621da 100644
@@ -6169,6 +6169,30 @@ bool current_is_workqueue_rescuer(void)
 	return worker && worker->rescue_wq;
 }
 
+/**
+ * current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker?
+ *
+ * Determine whether %current is a workqueue worker executing on a workqueue
+ * created with %WQ_MEM_RECLAIM.  This mirrors the condition that
+ * check_flush_dependency() warns on: flushing (or otherwise waiting on) a
+ * !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress
+ * guarantee and can deadlock.  Callers that may recurse into such a flush --
+ * e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own
+ * !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer
+ * the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline.
+ *
+ * Return: %true if %current is a %WQ_MEM_RECLAIM worker.  %false otherwise.
+ */
+bool current_is_workqueue_mem_reclaim(void)
+{
+	struct worker *worker = current_wq_worker();
+
+	return worker &&
+		((worker->current_pwq->wq->flags &
+		  (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM);
+}
+EXPORT_SYMBOL_GPL(current_is_workqueue_mem_reclaim);
+
 /**
  * workqueue_congested - test whether a workqueue is congested
  * @cpu: CPU in question
Signature · 1 line
2.44.0
#2 lore raw

PATCH 2/3 NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit

In reply to Mike Snitzer

nfs_local_commit() is reached only through nfs_initiate_commit(), and every path that supplies its "how" argument has already cleared FLUSH_SYNC:
__nfs_commit_inode() strips it (how &= ~FLUSH_SYNC) before dispatch and does its own waiting via wait_on_commit(), while the O_DIRECT path passes how=0. filelayout issues its DS commit with a NULL localio, so it never enters nfs_local_commit() at all. The FLUSH_SYNC branch has therefore been dead since it was introduced with commit 70ba381e1a43 ("nfs: add LOCALIO support").

Remove the never-taken FLUSH_SYNC branch along with the completion plumbing it was the sole user of: the struct nfs_local_fsync_ctx::done member, its initialization, and the complete() call in nfs_local_fsync_work(). With the branch gone the "how" parameter is unused, so drop it from nfs_local_commit() and its callers. No functional change.

Assisted-by Claude:claude-opus-4-8
---
 fs/nfs/internal.h |  4 ++--
 fs/nfs/localio.c  | 15 ++-------------
 fs/nfs/write.c    |  2 +-
 3 files changed, 5 insertions(+), 16 deletions(-)
fs/nfs/internal.h +2 -2
index 0338603e9674..66dbc9befdbb 100644
@@ -476,7 +476,7 @@ extern int nfs_local_doio(struct nfs_client *,
 			  const struct rpc_call_ops *);
 extern int nfs_local_commit(struct nfsd_file *,
 			    struct nfs_commit_data *,
-			    const struct rpc_call_ops *, int);
+			    const struct rpc_call_ops *);
 extern bool nfs_server_is_local(const struct nfs_client *clp);
 
 #else /* CONFIG_NFS_LOCALIO */
@@ -498,7 +498,7 @@ static inline int nfs_local_doio(struct nfs_client *clp,
 }
 static inline int nfs_local_commit(struct nfsd_file *localio,
 				struct nfs_commit_data *data,
-				const struct rpc_call_ops *call_ops, int how)
+				const struct rpc_call_ops *call_ops)
 {
 	return -EINVAL;
 }
fs/nfs/localio.c +2 -13
index d3e480888eb1..acbc2bddcf81 100644
@@ -52,7 +52,6 @@ struct nfs_local_fsync_ctx {
 	struct nfsd_file	*localio;
 	struct nfs_commit_data	*data;
 	struct work_struct	work;
-	struct completion	*done;
 };
 
 static bool localio_enabled __read_mostly = true;
@@ -1100,8 +1099,6 @@ nfs_local_fsync_work(struct work_struct *work)
 	status = nfs_local_run_commit(nfs_to->nfsd_file_file(ctx->localio),
 				      ctx->data);
 	nfs_local_commit_done(ctx->data, status);
-	if (ctx->done != NULL)
-		complete(ctx->done);
 	nfs_local_fsync_ctx_free(ctx);
 
 	current->flags = old_flags;
@@ -1117,14 +1114,13 @@ nfs_local_fsync_ctx_alloc(struct nfs_commit_data *data,
 		ctx->localio = localio;
 		ctx->data = data;
 		INIT_WORK(&ctx->work, nfs_local_fsync_work);
-		ctx->done = NULL;
 	}
 	return ctx;
 }
 
 int nfs_local_commit(struct nfsd_file *localio,
 		     struct nfs_commit_data *data,
-		     const struct rpc_call_ops *call_ops, int how)
+		     const struct rpc_call_ops *call_ops)
 {
 	struct nfs_local_fsync_ctx *ctx;
 
@@ -1136,14 +1132,7 @@ int nfs_local_commit(struct nfsd_file *localio,
 	}
 
 	nfs_local_init_commit(data, call_ops);
-
-	if (how & FLUSH_SYNC) {
-		DECLARE_COMPLETION_ONSTACK(done);
-		ctx->done = &done;
-		queue_work(nfslocaliod_workqueue, &ctx->work);
-		wait_for_completion(&done);
-	} else
-		queue_work(nfslocaliod_workqueue, &ctx->work);
+	queue_work(nfslocaliod_workqueue, &ctx->work);
 
 	return 0;
 }
fs/nfs/write.c +1 -1
index 0d7f2c2e599c..3afe243597fb 100644
@@ -1663,7 +1663,7 @@ int nfs_initiate_commit(struct rpc_clnt *clnt, struct nfs_commit_data *data,
 	dprintk("NFS: initiated commit call\n");
 
 	if (localio)
-		return nfs_local_commit(localio, data, call_ops, how);
+		return nfs_local_commit(localio, data, call_ops);
 
 	task = rpc_run_task(&task_setup_data);
 	if (IS_ERR(task))
Signature · 1 line
2.44.0
#3 lore raw

PATCH 3/3 NFS/localio: issue commit inline when not in a memory-reclaim context

In reply to Mike Snitzer

Extend the memory-reclaim-context test used for LOCALIO reads and writes to the commit (fsync) path. As with data IO, bouncing every commit through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue is only required when the submitting context is a memory-reclaim context: nfs_local_run_commit() calls vfs_fsync_range(), which may flush the underlying filesystem's own !WQ_MEM_RECLAIM workqueue, and doing so from a WQ_MEM_RECLAIM worker or a PF_MEMALLOC task trips check_flush_dependency().

The writeback path does exercise this: nfs_write_inode() (the ->write_inode super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq and reaches nfs_local_commit() via __nfs_commit_inode(), so that case must keep deferring. Application-context commits -- fsync (nfs_file_fsync), O_DIRECT (nfs_direct), and copy/clone (nfs42) -- are not in a reclaim context and now run the fsync inline via nfs_local_defer_io(), avoiding the per-commit workqueue hop.

Completion (nfs_commit_release_pages -> nfs_commit_end) then runs synchronously in the submitting context; higher layers already cope with this, as __nfs_commit_inode() dispatches the commit async and waits for it separately via wait_on_commit().

Assisted-by Claude:claude-opus-4-8
---
 fs/nfs/localio.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/nfs/localio.c +13 -1
index acbc2bddcf81..f42b6112a613 100644
@@ -1132,7 +1132,19 @@ int nfs_local_commit(struct nfsd_file *localio,
 	}
 
 	nfs_local_init_commit(data, call_ops);
-	queue_work(nfslocaliod_workqueue, &ctx->work);
+
+	/*
+	 * Run the commit (fsync) inline when not in a memory-reclaim context,
+	 * rather than bouncing through nfslocaliod_workqueue; see
+	 * nfs_local_defer_io().  Completion (nfs_commit_release_pages ->
+	 * nfs_commit_end) then runs synchronously, which higher layers cope
+	 * with: __nfs_commit_inode() dispatches async and waits via
+	 * wait_on_commit().
+	 */
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &ctx->work);
+	else
+		nfs_local_fsync_work(&ctx->work);
 
 	return 0;
 }
Signature · 1 line
2.44.0