nfs: Improve throughput for random buffered writes
Subsystems (2)
v2 cover nfs: Improve throughput for random buffered writes
[Resending because of messed up mailing list address]
Hello,
this is second revision of my patch series improving NFS throughput for buffered writes.
Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killable
Original cover letter below:
I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.
This patch series has two preparatory cleanups and then a patch implementing this idea.
Honza
Previous versions:
PATCH v2 1/3 nfs: Drop pointless check from nfs_commit_release_pages()
nfss->writeback is updated only when we are ending page writeback and at that moment we also clear nfss->write_congested. So there's no point in rechecking congestion state in nfs_commit_release_pages(). Drop the pointless check.
--- fs/nfs/write.c | 4 ---- 1 file changed, 4 deletions(-)
fs/nfs/write.c +0 -4
@@ -1837,7 +1837,6 @@ static void nfs_commit_release_pages(struct nfs_commit_data *data)struct nfs_page *req;int status = data->task.tk_status;struct nfs_commit_info cinfo; -struct nfs_server *nfss;struct folio *folio;while (!list_empty (&data->pages)) {
@@ -1880,9 +1879,6 @@ static void nfs_commit_release_pages(struct nfs_commit_data *data)/* Latency breaker */ cond_resched (); } - nfss =NFS_SERVER (data->inode); -if (atomic_long_read (&nfss->writeback) <NFS_CONGESTION_OFF_THRESH ) - nfss->write_congested =0 ;nfs_init_cinfo (&cinfo, data->inode, data->dreq);nfs_commit_end (cinfo.mds);
Signature · 1 line
2.35.3
PATCH v2 2/3 nfs: Properly initialize server->writeback
In reply to Jan KaraAtomic types should better be initialized with atomic_long_set() instead of relying on zeroing done by kzalloc(). Clean this up.
--- fs/nfs/client.c | 2 ++ 1 file changed, 2 insertions(+)
fs/nfs/client.c +2 -0
@@ -994,6 +994,8 @@ struct nfs_server *nfs_alloc_server(void) server->change_attr_type =NFS4_CHANGE_TYPE_IS_UNDEFINED ; +atomic_long_set (&server->writeback,0 ); +ida_init (&server->openowner_id);ida_init (&server->lockowner_id);pnfs_init_server (server);
Signature · 1 line
2.35.3
Re: [PATCH v2 2/3] nfs: Properly initialize server->writeback
Quoted 20 lines · Jan Kara
Atomic types should better be initialized with
atomic_long_set()instead
of relying on zeroing done bykzalloc(). Clean this up.Reviewed-by Sagi Grimberg <[email protected]>Signed-off-by Jan Kara <[email protected]>---
fs/nfs/client.c | 2 ++1 file changed, 2 insertions(+)fs/nfs/client.c +0
-0@@ -994,6 +994,8 @@ struct nfs_server *nfs_alloc_server(void)
server->change_attr_type=NFS4_CHANGE_TYPE_IS_UNDEFINED;fs/nfs/client.c +2
-0+atomic_long_set (&server->writeback,0 ); +
Quoted 3 lines
ida_init(&server->openowner_id);
ida_init(&server->lockowner_id);
pnfs_init_server(server);
It seems a little pointless, but ok.
PATCH v2 3/3 nfs: Block on write congestion
In reply to Jan KaraCommit 6df25e58532b ("nfs: remove reliance on bdi congestion") introduced NFS-private solution for limiting number of writes outstanding against a particular server. Unlike previous bdi congestion this algorithm actually works and limits number of outstanding writeback pages to nfs_congestion_kb which scales with amount of client's memory and is capped at 256 MB. As a result some workloads such as random buffered writes over NFS got slower (from ~170 MB/s to ~126 MB/s). The fio command to reproduce is:
fio --direct=0 --ioengine=sync --thread --invalidate=1 --group_reporting=1
--runtime=300 --fallocate=posix --ramp_time=10 --new_group --rw=randwrite --size=64256m --numjobs=4 --bs=4k --fsync_on_close=1 --end_fsync=1
This happens because the client sends ~256 MB worth of dirty pages to the server and any further background writeback request is ignored until the number of writeback pages gets below the threshold of 192 MB. By the time this happens and clients decides to trigger another round of writeback, the server often has no pages to write and the disk is idle.
To fix this problem and make the client react faster to eased congestion of the server by blocking waiting for congestion to resolve instead of aborting writeback. This improves the random 4k buffered write throughput to 184 MB/s.
--- fs/nfs/client.c | 1 + fs/nfs/write.c | 15 +++++++++++---- include/linux/nfs_fs_sb.h | 1 + 3 files changed, 13 insertions(+), 4 deletions(-)
fs/nfs/client.c +1 -0
@@ -994,6 +994,7 @@ struct nfs_server *nfs_alloc_server(void) server->change_attr_type =NFS4_CHANGE_TYPE_IS_UNDEFINED ; +init_waitqueue_head (&server->write_congestion_wait);atomic_long_set (&server->writeback,0 );ida_init (&server->openowner_id);
fs/nfs/write.c +11 -4
@@ -423,8 +423,10 @@ static void nfs_folio_end_writeback(struct folio *folio)folio_end_writeback (folio);if (atomic_long_dec_return (&nfss->writeback) < -NFS_CONGESTION_OFF_THRESH ) +NFS_CONGESTION_OFF_THRESH ) { nfss->write_congested =0 ; +wake_up_all (&nfss->write_congestion_wait); + } }static void nfs_page_end_writeback (struct nfs_page *req)
@@ -698,12 +700,17 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)struct nfs_pageio_descriptor pgio;struct nfs_io_completion *ioc =NULL ;unsigned int mntflags =NFS_SERVER (inode)->flags; +struct nfs_server *nfss =NFS_SERVER (inode);int priority =0 ;int err; -if (wbc->sync_mode ==WB_SYNC_NONE && -NFS_SERVER (inode)->write_congested) -return 0 ; +/* Wait with writeback until write congestion eases */ +if (wbc->sync_mode ==WB_SYNC_NONE && nfss->write_congested) { + err =wait_event_killable (nfss->write_congestion_wait, + nfss->write_congested ==0 ); +if (err) +return err; + }nfs_inc_stats (inode,NFSIOS_VFSWRITEPAGES );
include/linux/nfs_fs_sb.h +1 -0
@@ -140,6 +140,7 @@ struct nfs_server {struct rpc_clnt * client_acl;/* ACL RPC client handle */ struct nlm_host *nlm_host;/* NLM client handle */ struct nfs_iostats __percpu *io_stats;/* I/O statistics */ +wait_queue_head_t write_congestion_wait;/* wait until write congestion eases */ atomic_long_t writeback;/* number of writeback pages */ unsigned int write_congested;/* flag set when writeback gets too high */ unsigned int flags;/* various flags */
Signature · 1 line
2.35.3
Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?
Honza
Quoted 29 lines · Jan Kara
[Resending because of messed up mailing list address]
Hello,
this is second revision of my patch series improving NFS throughput for buffered writes.
Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killableOriginal cover letter below:
I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.
This patch series has two preparatory cleanups and then a patch implementing this idea.
Honza
Previous versions:
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Anna Schumaker
maintainerHi Jan,
Quoted 4 lines · Jan Kara
Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?
I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14
I hope this helps!
Anna
Quoted 36 lines
Honza
Quoted 21 lines · Jan Kara
[Resending because of messed up mailing list address]
Hello,
this is second revision of my patch series improving NFS throughput for buffered writes.
Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killableOriginal cover letter below:
I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.
Quoted 8 lines
This patch series has two preparatory cleanups and then a patch implementing this idea.
Honza
Previous versions:
--
Jan Kara <[email protected]>
SUSE Labs, CR
Hi Anna!
Quoted 7 lines · Anna Schumaker
Quoted 3 lines · Jan Kara
Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?
I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14
Aha, great! I was checking the tree mentioned in MAINTAINERS file which is Trond's one and because it seemed fairly active it didn't occur to me you are maintaining your tree as well. Thanks for the link!
One more question: Do you plan to push these changes for 6.11 or 6.12?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Anna Schumaker
maintainerQuoted 9 lines · Jan Kara
Hi Anna!
Quoted 5 lines · Anna Schumaker
Quoted 3 lines · Jan Kara
Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?
Quoted 6 lines
Quoted 2 lines
I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14Aha, great! I was checking the tree mentioned in
MAINTAINERSfile which is Trond's one and because it seemed fairly active it didn't occur to me you are maintaining your tree as well. Thanks for the link!
And this is the first time I've thought about adding my tree to the MAINTAINERS file. I should probably do that to avoid confusion in the future!
Quoted 2 lines
One more question: Do you plan to push these changes for 6.11 or 6.12?
They'll be in my pull request for 6.11 that I hope to have out in the next hour or so.
Anna
Quoted 5 lines
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR