NFS/localio: issue IO inline when not reclaiming memory
Subsystems (3)
cover NFS/localio: issue IO inline when not reclaiming memory
Hi,
Every NFS LOCALIO read, write and commit is currently issued indirectly: it is queued onto the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than being submitted to the underlying filesystem in the calling context. That intermediate hop was introduced by commit b9f5dd57f4a5 ("nfs/localio:
use dedicated workqueues for filesystem read and write") because:
LOCALIO submits IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing so from a WQ_MEM_RECLAIM worker -- most importantly writeback's wb_workfn running on bdi_wq -- or from an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock.
However, that hazard only exists when the submitting context is itself a memory-reclaim context. For ordinary application/task submission -- e.g. O_DIRECT, or an fsync-driven commit -- the workqueue hop buys nothing: it just adds a context switch and scheduling latency per IO and throws away the NFS client's inherent application-context parallelism.
This series makes the hop conditional. It adds a small workqueue-core helper, current_is_workqueue_mem_reclaim(), that reports whether %current is a WQ_MEM_RECLAIM worker using exactly the predicate check_flush_dependency() warns on. LOCALIO uses it (together with the existing PF_MEMALLOC test) in a new nfs_local_defer_io() helper to decide per-IO whether it must defer to nfslocaliod_workqueue or may issue the IO inline. Reclaim contexts still defer and are unaffected; everything else runs inline.
Patch 1 adds current_is_workqueue_mem_reclaim() and applies the gating to
the read and write paths.
Patch 2 removes never-taken FLUSH_SYNC handling from nfs_local_commit()
(every caller supplies a FLUSH_SYNC-stripped "how"), which also drops the sole user of the ctx->done completion plumbing and the now-unused "how" argument. No functional change.
Patch 3 extends the same gating to the commit (fsync) path. Note the
writeback-triggered commit does reach here in reclaim context -- nfs_write_inode() (the ->write_inode super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq -- so that case correctly keeps deferring; only app-context commits run inline.
Patch 1 touches the workqueue core (kernel/workqueue.c, include/linux/ workqueue.h), hence the Cc to Tejun Heo, Lai Jiangshan and LKML. The new export mirrors the existing current_is_workqueue_rescuer()/current_work() context-introspection helpers.
Additional note for reviewers:
- The inline path now stacks the NFS pgio path plus the underlying filesystem's ->write_iter/->read_iter (and vfs_fsync_range) in task context. The reclaim-context stack-depth concern of b9f5dd57f4a5 (2) is avoided for the deferred paths; feedback on inline stack headroom under deeply-stacking filesystems (XFS especially) is welcome.
All review appreciated, thanks.
Mike
Mike Snitzer (3):
NFS/localio: issue IO inline when not in a memory-reclaim context NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit NFS/localio: issue commit inline when not in a memory-reclaim context
fs/nfs/internal.h | 4 +-- fs/nfs/localio.c | 56 ++++++++++++++++++++++++++++++--------- fs/nfs/write.c | 2 +- include/linux/workqueue.h | 1 + kernel/workqueue.c | 24 +++++++++++++++++ 5 files changed, 71 insertions(+), 16 deletions(-)
Signature · 1 line
2.44.0
PATCH 1/3 NFS/localio: issue IO inline when not in a memory-reclaim context
Every LOCALIO read and write is currently bounced through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually required when the submitting context is a memory-reclaim context: LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock, which is why commit b9f5dd57f4a5 ("nfs/localio: use dedicated workqueues for filesystem read and write") introduced the intermediate workqueue.
Outside of reclaim context -- ordinary application/task submission such as O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and merely adds a context switch and scheduling latency per IO while discarding the NFS client's inherent application-context parallelism.
Add current_is_workqueue_mem_reclaim(), which reports whether %current is a WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency() warns on. Use it, together with the PF_MEMALLOC check, in the new nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read() and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue the IO inline. Buffered writeback continues to bounce (wb_workfn is a WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.
Running nfs_local_call_write() inline is safe: it already saves and restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it sets and scopes the file opener's creds. The async O_DIRECT completion path is likewise unaffected: when the underlying filesystem returns -EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete / nfs_local_write_aio_complete) can run in bottom-half context and so must still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is independent of how the IO was submitted, and this change leaves it as-is; only the submission side stops unconditionally hopping through nfslocaliod_workqueue.
--- fs/nfs/localio.c | 33 +++++++++++++++++++++++++++++++-- include/linux/workqueue.h | 1 + kernel/workqueue.c | 24 ++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 2 deletions(-)
fs/nfs/localio.c +31 -2
@@ -699,6 +699,29 @@ static void nfs_local_call_read(struct work_struct *work) } } +/* +* Decide whether LOCALIO must defer submission to the dedicated +* !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than issue the IO inline. +* +* LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS), +* which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing so from a +* memory-reclaim context -- either a WQ_MEM_RECLAIM worker (most importantly +* writeback's wb_workfn running on bdi_wq) or an explicit reclaim task +* (PF_MEMALLOC) -- would trip check_flush_dependency() and risks a +* forward-progress deadlock; see commit b9f5dd57f4a5 ("nfs/localio: use +* dedicated workqueues for filesystem read and write"). In that case defer +* to nfslocaliod_workqueue. +* +* Otherwise (ordinary application/task context, e.g. O_DIRECT or fsync-driven +* submission) issue the IO inline: this preserves the NFS client's inherent +* application-context parallelism and avoids the per-IO workqueue hop. +*/ +static inline bool nfs_local_defer_io (void ) +{ +return (current->flags &PF_MEMALLOC ) || +current_is_workqueue_mem_reclaim (); +} +static void nfs_local_do_read (struct nfs_local_kiocb *iocb,const struct rpc_call_ops *call_ops) {
@@ -711,7 +734,10 @@ static void nfs_local_do_read(struct nfs_local_kiocb *iocb, hdr->res.eof = false;INIT_WORK (&iocb->work, nfs_local_call_read); -queue_work (nfslocaliod_workqueue, &iocb->work); +if (nfs_local_defer_io ()) +queue_work (nfslocaliod_workqueue, &iocb->work); +else +nfs_local_call_read (&iocb->work); } static void
@@ -929,7 +955,10 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb,nfs_set_local_verifier (hdr->inode, hdr->res.verf, hdr->args.stable);INIT_WORK (&iocb->work, nfs_local_call_write); -queue_work (nfslocaliod_workqueue, &iocb->work); +if (nfs_local_defer_io ()) +queue_work (nfslocaliod_workqueue, &iocb->work); +else +nfs_local_call_write (&iocb->work); } static struct nfs_local_kiocb *
include/linux/workqueue.h +1 -0
@@ -659,6 +659,7 @@ extern void workqueue_set_min_active(struct workqueue_struct *wq, int min_active);extern struct work_struct *current_work (void );extern bool current_is_workqueue_rescuer (void ); +extern bool current_is_workqueue_mem_reclaim (void );extern bool workqueue_congested (int cpu,struct workqueue_struct *wq);extern unsigned int work_busy (struct work_struct *work); extern __printf(1, 2) void set_worker_desc(const char *fmt, ...);
kernel/workqueue.c +24 -0
@@ -6169,6 +6169,30 @@ bool current_is_workqueue_rescuer(void)return worker && worker->rescue_wq; } +/** +* current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker? +* +* Determine whether %current is a workqueue worker executing on a workqueue +* created with %WQ_MEM_RECLAIM. This mirrors the condition that +* check_flush_dependency() warns on: flushing (or otherwise waiting on) a +* !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress +* guarantee and can deadlock. Callers that may recurse into such a flush -- +* e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own +* !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer +* the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline. +* +* Return: %true if %current is a %WQ_MEM_RECLAIM worker. %false otherwise. +*/ +bool current_is_workqueue_mem_reclaim (void ) +{ +struct worker *worker =current_wq_worker (); + +return worker && + ((worker->current_pwq->wq->flags & + (WQ_MEM_RECLAIM | __WQ_LEGACY)) ==WQ_MEM_RECLAIM ); +} +EXPORT_SYMBOL_GPL (current_is_workqueue_mem_reclaim); +/** * workqueue_congested - test whether a workqueue is congested * @cpu: CPU in question
Signature · 1 line
2.44.0
PATCH 2/3 NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit
In reply to Mike Snitzernfs_local_commit() is reached only through nfs_initiate_commit(), and every path that supplies its "how" argument has already cleared FLUSH_SYNC:__nfs_commit_inode() strips it (how &= ~FLUSH_SYNC) before dispatch and does its own waiting via wait_on_commit(), while the O_DIRECT path passes how=0. filelayout issues its DS commit with a NULL localio, so it never enters nfs_local_commit() at all. The FLUSH_SYNC branch has therefore been dead since it was introduced with commit 70ba381e1a43 ("nfs: add LOCALIO support").
Remove the never-taken FLUSH_SYNC branch along with the completion plumbing it was the sole user of: the struct nfs_local_fsync_ctx::done member, its initialization, and the complete() call in nfs_local_fsync_work(). With the branch gone the "how" parameter is unused, so drop it from nfs_local_commit() and its callers. No functional change.
--- fs/nfs/internal.h | 4 ++-- fs/nfs/localio.c | 15 ++------------- fs/nfs/write.c | 2 +- 3 files changed, 5 insertions(+), 16 deletions(-)
fs/nfs/internal.h +2 -2
@@ -476,7 +476,7 @@ extern int nfs_local_doio(struct nfs_client *, const struct rpc_call_ops *);extern int nfs_local_commit (struct nfsd_file *,struct nfs_commit_data *, -const struct rpc_call_ops *,int ); +const struct rpc_call_ops *);extern bool nfs_server_is_local (const struct nfs_client *clp);#else /* CONFIG_NFS_LOCALIO */
@@ -498,7 +498,7 @@ static inline int nfs_local_doio(struct nfs_client *clp, }static inline int nfs_local_commit (struct nfsd_file *localio,struct nfs_commit_data *data, -const struct rpc_call_ops *call_ops,int how) +const struct rpc_call_ops *call_ops) {return -EINVAL ; }
fs/nfs/localio.c +2 -13
@@ -52,7 +52,6 @@ struct nfs_local_fsync_ctx {struct nfsd_file *localio;struct nfs_commit_data *data;struct work_struct work; -struct completion *done; };static bool localio_enabled__read_mostly = true;
@@ -1100,8 +1099,6 @@ nfs_local_fsync_work(struct work_struct *work) status =nfs_local_run_commit (nfs_to->nfsd_file_file (ctx->localio), ctx->data);nfs_local_commit_done (ctx->data, status); -if (ctx->done !=NULL ) -complete (ctx->done);nfs_local_fsync_ctx_free (ctx); current->flags = old_flags;
@@ -1117,14 +1114,13 @@ nfs_local_fsync_ctx_alloc(struct nfs_commit_data *data, ctx->localio = localio; ctx->data = data;INIT_WORK (&ctx->work, nfs_local_fsync_work); - ctx->done =NULL ; }return ctx; }int nfs_local_commit (struct nfsd_file *localio,struct nfs_commit_data *data, -const struct rpc_call_ops *call_ops,int how) +const struct rpc_call_ops *call_ops) {struct nfs_local_fsync_ctx *ctx;
@@ -1136,14 +1132,7 @@ int nfs_local_commit(struct nfsd_file *localio, }nfs_local_init_commit (data, call_ops); - -if (how &FLUSH_SYNC ) { -DECLARE_COMPLETION_ONSTACK (done); - ctx->done = &done; -queue_work (nfslocaliod_workqueue, &ctx->work); -wait_for_completion (&done); - }else -queue_work (nfslocaliod_workqueue, &ctx->work); +queue_work (nfslocaliod_workqueue, &ctx->work);return 0 ; }
fs/nfs/write.c +1 -1
@@ -1663,7 +1663,7 @@ int nfs_initiate_commit(struct rpc_clnt *clnt, struct nfs_commit_data *data,dprintk ("NFS: initiated commit call\n" );if (localio) -return nfs_local_commit (localio, data, call_ops, how); +return nfs_local_commit (localio, data, call_ops); task =rpc_run_task (&task_setup_data);if (IS_ERR (task))
Signature · 1 line
2.44.0
PATCH 3/3 NFS/localio: issue commit inline when not in a memory-reclaim context
In reply to Mike SnitzerExtend the memory-reclaim-context test used for LOCALIO reads and writes to the commit (fsync) path. As with data IO, bouncing every commit through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue is only required when the submitting context is a memory-reclaim context: nfs_local_run_commit() calls vfs_fsync_range(), which may flush the underlying filesystem's own !WQ_MEM_RECLAIM workqueue, and doing so from a WQ_MEM_RECLAIM worker or a PF_MEMALLOC task trips check_flush_dependency().
The writeback path does exercise this: nfs_write_inode() (the ->write_inode super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq and reaches nfs_local_commit() via __nfs_commit_inode(), so that case must keep deferring. Application-context commits -- fsync (nfs_file_fsync), O_DIRECT (nfs_direct), and copy/clone (nfs42) -- are not in a reclaim context and now run the fsync inline via nfs_local_defer_io(), avoiding the per-commit workqueue hop.
Completion (nfs_commit_release_pages -> nfs_commit_end) then runs synchronously in the submitting context; higher layers already cope with this, as __nfs_commit_inode() dispatches the commit async and waits for it separately via wait_on_commit().
--- fs/nfs/localio.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
fs/nfs/localio.c +13 -1
@@ -1132,7 +1132,19 @@ int nfs_local_commit(struct nfsd_file *localio, }nfs_local_init_commit (data, call_ops); -queue_work (nfslocaliod_workqueue, &ctx->work); + +/* +* Run the commit (fsync) inline when not in a memory-reclaim context, +* rather than bouncing through nfslocaliod_workqueue; see +* nfs_local_defer_io(). Completion (nfs_commit_release_pages -> +* nfs_commit_end) then runs synchronously, which higher layers cope +* with: __nfs_commit_inode() dispatches async and waits via +* wait_on_commit(). +*/ +if (nfs_local_defer_io ()) +queue_work (nfslocaliod_workqueue, &ctx->work); +else +nfs_local_fsync_work (&ctx->work);return 0 ; }
Signature · 1 line
2.44.0