Skip to contentSkip to conversation

nfs: Improve throughput for random buffered writes

PATCH RESEND In review Jan Kara 3 participants 9 messages Jul 1, 2024 – Jul 18, 2024

Subsystems (2)
lore
Share & tools
lore

Jan Kara

author
#0 lore raw

v2 cover nfs: Improve throughput for random buffered writes

[Resending because of messed up mailing list address]

Hello,

this is second revision of my patch series improving NFS throughput for buffered writes.

Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killable

Original cover letter below:

I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.

This patch series has two preparatory cleanups and then a patch implementing this idea.

Honza

Previous versions:

author
#1 lore raw

PATCH v2 1/3 nfs: Drop pointless check from nfs_commit_release_pages()

nfss->writeback is updated only when we are ending page writeback and at that moment we also clear nfss->write_congested. So there's no point in rechecking congestion state in nfs_commit_release_pages(). Drop the pointless check.

---
 fs/nfs/write.c | 4 ----
 1 file changed, 4 deletions(-)
fs/nfs/write.c +0 -4
index 5fc12a721ec3..c6255d7edd3c 100644
@@ -1837,7 +1837,6 @@ static void nfs_commit_release_pages(struct nfs_commit_data *data)
 	struct nfs_page	*req;
 	int status = data->task.tk_status;
 	struct nfs_commit_info cinfo;
-	struct nfs_server *nfss;
 	struct folio *folio;
 
 	while (!list_empty(&data->pages)) {
@@ -1880,9 +1879,6 @@ static void nfs_commit_release_pages(struct nfs_commit_data *data)
 		/* Latency breaker */
 		cond_resched();
 	}
-	nfss = NFS_SERVER(data->inode);
-	if (atomic_long_read(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH)
-		nfss->write_congested = 0;
 
 	nfs_init_cinfo(&cinfo, data->inode, data->dreq);
 	nfs_commit_end(cinfo.mds);
Signature · 1 line
2.35.3
author
#2 lore raw

PATCH v2 2/3 nfs: Properly initialize server->writeback

In reply to Jan Kara

Atomic types should better be initialized with atomic_long_set() instead of relying on zeroing done by kzalloc(). Clean this up.

---
 fs/nfs/client.c | 2 ++
 1 file changed, 2 insertions(+)
fs/nfs/client.c +2 -0
index de77848ae654..3b252dceebf5 100644
@@ -994,6 +994,8 @@ struct nfs_server *nfs_alloc_server(void)
 
 	server->change_attr_type = NFS4_CHANGE_TYPE_IS_UNDEFINED;
 
+	atomic_long_set(&server->writeback, 0);
+
 	ida_init(&server->openowner_id);
 	ida_init(&server->lockowner_id);
 	pnfs_init_server(server);
Signature · 1 line
2.35.3
maintainer Acked-by on v2 2/3
#3 lore raw

Re: [PATCH v2 2/3] nfs: Properly initialize server->writeback

Quoted 20 lines · Jan Kara

Atomic types should better be initialized with atomic_long_set() instead
of relying on zeroing done by kzalloc(). Clean this up.

---
 fs/nfs/client.c | 2 ++

 1 file changed, 2 insertions(+)
fs/nfs/client.c +0 -0
index de77848ae654..3b252dceebf5 100644
@@ -994,6 +994,8 @@ struct nfs_server *nfs_alloc_server(void)


  server->change_attr_type = NFS4_CHANGE_TYPE_IS_UNDEFINED;

fs/nfs/client.c +2 -0
+	atomic_long_set(&server->writeback, 0);
+
Quoted 3 lines

  ida_init(&server->openowner_id);
  ida_init(&server->lockowner_id);
  pnfs_init_server(server);

It seems a little pointless, but ok.

author
#4 lore raw

PATCH v2 3/3 nfs: Block on write congestion

In reply to Jan Kara

Commit 6df25e58532b ("nfs: remove reliance on bdi congestion") introduced NFS-private solution for limiting number of writes outstanding against a particular server. Unlike previous bdi congestion this algorithm actually works and limits number of outstanding writeback pages to nfs_congestion_kb which scales with amount of client's memory and is capped at 256 MB. As a result some workloads such as random buffered writes over NFS got slower (from ~170 MB/s to ~126 MB/s). The fio command to reproduce is:

fio --direct=0 --ioengine=sync --thread --invalidate=1 --group_reporting=1
--runtime=300 --fallocate=posix --ramp_time=10 --new_group --rw=randwrite --size=64256m --numjobs=4 --bs=4k --fsync_on_close=1 --end_fsync=1

This happens because the client sends ~256 MB worth of dirty pages to the server and any further background writeback request is ignored until the number of writeback pages gets below the threshold of 192 MB. By the time this happens and clients decides to trigger another round of writeback, the server often has no pages to write and the disk is idle.

To fix this problem and make the client react faster to eased congestion of the server by blocking waiting for congestion to resolve instead of aborting writeback. This improves the random 4k buffered write throughput to 184 MB/s.

---
 fs/nfs/client.c           |  1 +
 fs/nfs/write.c            | 15 +++++++++++----
 include/linux/nfs_fs_sb.h |  1 +
 3 files changed, 13 insertions(+), 4 deletions(-)
fs/nfs/client.c +1 -0
index 3b252dceebf5..8286edd6062d 100644
@@ -994,6 +994,7 @@ struct nfs_server *nfs_alloc_server(void)
 
 	server->change_attr_type = NFS4_CHANGE_TYPE_IS_UNDEFINED;
 
+	init_waitqueue_head(&server->write_congestion_wait);
 	atomic_long_set(&server->writeback, 0);
 
 	ida_init(&server->openowner_id);
fs/nfs/write.c +11 -4
index c6255d7edd3c..d5f8f77a2352 100644
@@ -423,8 +423,10 @@ static void nfs_folio_end_writeback(struct folio *folio)
 
 	folio_end_writeback(folio);
 	if (atomic_long_dec_return(&nfss->writeback) <
-	    NFS_CONGESTION_OFF_THRESH)
+	    NFS_CONGESTION_OFF_THRESH) {
 		nfss->write_congested = 0;
+		wake_up_all(&nfss->write_congestion_wait);
+	}
 }
 
 static void nfs_page_end_writeback(struct nfs_page *req)
@@ -698,12 +700,17 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 	struct nfs_pageio_descriptor pgio;
 	struct nfs_io_completion *ioc = NULL;
 	unsigned int mntflags = NFS_SERVER(inode)->flags;
+	struct nfs_server *nfss = NFS_SERVER(inode);
 	int priority = 0;
 	int err;
 
-	if (wbc->sync_mode == WB_SYNC_NONE &&
-	    NFS_SERVER(inode)->write_congested)
-		return 0;
+	/* Wait with writeback until write congestion eases */
+	if (wbc->sync_mode == WB_SYNC_NONE && nfss->write_congested) {
+		err = wait_event_killable(nfss->write_congestion_wait,
+					  nfss->write_congested == 0);
+		if (err)
+			return err;
+	}
 
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES);
 
include/linux/nfs_fs_sb.h +1 -0
index 92de074e63b9..38a128796a76 100644
@@ -140,6 +140,7 @@ struct nfs_server {
 	struct rpc_clnt *	client_acl;	/* ACL RPC client handle */
 	struct nlm_host		*nlm_host;	/* NLM client handle */
 	struct nfs_iostats __percpu *io_stats;	/* I/O statistics */
+	wait_queue_head_t	write_congestion_wait;	/* wait until write congestion eases */
 	atomic_long_t		writeback;	/* number of writeback pages */
 	unsigned int		write_congested;/* flag set when writeback gets too high */
 	unsigned int		flags;		/* various flags */
Signature · 1 line
2.35.3
author
#5 lore raw
In reply to Jan Kara

Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?

Honza

Quoted 29 lines · Jan Kara

[Resending because of messed up mailing list address]

Hello,

this is second revision of my patch series improving NFS throughput for buffered writes.

Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killable

Original cover letter below:

I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.

This patch series has two preparatory cleanups and then a patch implementing this idea.

Honza

Previous versions:

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#6 lore raw

Hi Jan,

Quoted 4 lines · Jan Kara

Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?

I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14

I hope this helps!
Anna

Quoted 36 lines

Honza

Quoted 21 lines · Jan Kara

[Resending because of messed up mailing list address]

Hello,

this is second revision of my patch series improving NFS throughput for buffered writes.

Changes since v1:
* Added Reviewed-by tags
* Made sleep waiting for congestion to resolve killable

Original cover letter below:

I was thinking how to best address the performance regression coming from NFS write congestion. After considering various options and concerns raised in the previous discussion, I've got an idea for a simple option that could help to keep the server more busy - just mimick what block devices do and block the flush worker waiting for congestion to resolve instead of aborting the writeback. And it actually helps enough that I don't think more complex solutions are warranted at this point.

Quoted 8 lines

This patch series has two preparatory cleanups and then a patch implementing this idea.

Honza

Previous versions:

--
Jan Kara <[email protected]>
SUSE Labs, CR

author
#7 lore raw

Hi Anna!

Quoted 7 lines · Anna Schumaker
Quoted 3 lines · Jan Kara

Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?

I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14

Aha, great! I was checking the tree mentioned in MAINTAINERS file which is Trond's one and because it seemed fairly active it didn't occur to me you are maintaining your tree as well. Thanks for the link!

One more question: Do you plan to push these changes for 6.11 or 6.12?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#8 lore raw
Quoted 9 lines · Jan Kara

Hi Anna!

Quoted 5 lines · Anna Schumaker
Quoted 3 lines · Jan Kara

Ping? I don't see these patches being in NFS git tree. Did they fall through the cracks?

Quoted 6 lines
Quoted 2 lines

I have these patches in my tree starting with this commit:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=commit;h=37d4159dd25ade59ce0fecc75984240e5f7abc14

Aha, great! I was checking the tree mentioned in MAINTAINERS file which is Trond's one and because it seemed fairly active it didn't occur to me you are maintaining your tree as well. Thanks for the link!

And this is the first time I've thought about adding my tree to the MAINTAINERS file. I should probably do that to avoid confusion in the future!

Quoted 2 lines

One more question: Do you plan to push these changes for 6.11 or 6.12?

They'll be in my pull request for 6.11 that I hope to have out in the next hour or so.

Anna

Quoted 5 lines

Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR