nfsd write throughput

fs/nfsd/vfs.c +9 -0

@@ -810,6 +811,22 @@
 		}
 		last_ino = inode->i_ino;
 		last_dev = inode->i_sb->s_dev;
+	} else if (err >= 0 && !stable) {
+		/* If we've been writing several pages, schedule them
+		 * for the disk immediately. The client may be streaming
+		 * and we don't want to hang on a huge journal sync when the
+		 * commit comes in
+		 */
+		struct address_space	*mapping;
+
+		/* This assumes a minimum page size of 1K, and will issue

fs/nfsd/vfs.c line 826

Quoted 2 lines

fs/nfsd/vfs.c +2 -0

+		if ((cnt & 1023) == 0
+		 && ((offset / cnt) & 63) == 0

First, the way the v3 server is supposed to work is that normal page cache pressure pushes pages from unstable writes to disk before the COMMIT call arrives from the client. The best way to achieve this for a dedicated NFS server box is tuning the pdflush parameters to be more aggressive about writing back dirty pages, e.g. bumping down the following in /proc/vm: dirty_background_ratio, dirty_ratio, dirty_writeback_centisecs, and dirty_expire_centisecs. I have to

I think another useful approach would be to writeback pages which have been written by NFS unstable writes at a faster rate than pages written by local applications, i.e. add a new /proc/vm/ sysctl like nfs_dirty_writeback_centisecs and a per-page flag.

For example, imagine the disk backend is a hardware RAID5 with a stripe size of 128K or greater and the client is doing streaming 32K WRITE calls. With your patch, every second WRITE call will now try to write half a RAID stripe unit,

size. Whether the page cache and fs actually do the right thing is another matter, but that's where the responsibility lies.

Quoted 7 lines · Greg Banks

First, the way the v3 server is supposed to work is that normal page cache pressure pushes pages from unstable writes to disk before the COMMIT call arrives from the client. The best way to achieve this for a dedicated NFS server box is tuning the pdflush parameters to be more aggressive about writing back dirty pages, e.g. bumping down the following in /proc/vm: dirty_background_ratio, dirty_ratio, dirty_writeback_centisecs, and dirty_expire_centisecs. I have to

Yes and no. Can we expect every user to fiddle with the pdflush tunables to get an NFS server that performs reasonably well?

Quoted 4 lines

I think another useful approach would be to writeback pages which have been written by NFS unstable writes at a faster rate than pages written by local applications, i.e. add a new /proc/vm/ sysctl like nfs_dirty_writeback_centisecs and a per-page flag.

That may be a useful solution, too. My patch basically does what fadvise(WONTNEED) does.

Quoted 4 lines

For example, imagine the disk backend is a hardware RAID5 with a stripe size of 128K or greater and the client is doing streaming 32K WRITE calls. With your patch, every second WRITE call will now try to write half a RAID stripe unit,

No, it doesn't. If you look at the if() expression, you'll see it writes every 64 client-size pages.

Quoted 2 lines

fs/nfsd/vfs.c line 826

Quoted 2 lines

fs/nfsd/vfs.c +2 -0

+		if ((cnt & 1023) == 0
+		 && ((offset / cnt) & 63) == 0

In the worst case that's every 64K,
but for Linux clients that's every 256K which is a reasonable size for IDE DMA, as well as most RAID configurations

I think another useful approach would be to writeback pages which have been written by NFS unstable writes at a faster rate than pages written by local applications, i.e. add a new /proc/vm/ sysctl like nfs_dirty_writeback_centisecs and a per-page flag.

Quoted 2 lines

That may be a useful solution, too. My patch basically does what fadvise(WONTNEED) does.

Sure, the key question is when and for how many pages. You don't really have enough information in nfsd_write() to tell that safely.

It writes every time offset' is a multiple of 64 times cnt' and cnt' is a multiple of 1024. At this point cnt' is the length of the data received in the WRITE call, which has only a vague relationship to the client page size.

Quoted 4 lines · Greg Banks

I think another useful approach would be to writeback pages which have been written by NFS unstable writes at a faster rate than pages written by local applications, i.e. add a new /proc/vm/ sysctl like nfs_dirty_writeback_centisecs and a per-page flag.

The problem with this approach is that we have no access to the pages. nfsd_write goes through writev.

Quoted 5 lines

Quoted 2 lines

That may be a useful solution, too. My patch basically does what fadvise(WONTNEED) does.

Sure, the key question is when and for how many pages. You don't really have enough information in nfsd_write() to tell that safely.

Well, for streaming writes "everything we've written so far" is a reasonable approximation. Random writes may receive a penalty, I admit.

Quoted 4 lines

It writes every time offset' is a multiple of 64 times cnt' and cnt' is a multiple of 1024. At this point cnt' is the length of the data received in the WRITE call, which has only a vague relationship to the client page size.

Okay, I should have been more precise: the test tries to make sure we're seeing a full wsize worth of data, which is usually a good indication of writes being streamed, and tries to lump enough of them together to allow the file system to make an intelligent decision.

I'm not claiming this is god's wisdom - I'm trying out ideas :)

I'm wondering what could happen if we change nfsd_sync to not take the i_sem at all... I'll talk to a few VFS folks around here and try to find out.

Hi folks,

I've been looking at the problem from a different angle...

Theory:
The main bottleneck is that we spend a long time in commit(), blocking other WRITE calls from making any progress (thereby stalling all NFS clients). The reason is what we take inode->i_sem in nfsd_sync, but the writev() code wants to grab the same semaphore.

Circumstantial Evidence:

I've been doing some tests with the latencies of WRITE and COMMIT, using a single stream write. The average time we spend in nfsd_write is miniscule, usually it's less than 2 milliseconds. However when a commit comes in, we take a hit there as well - something around 500 ms for reiser, and 400 ms for ext3. Syncing to reiser frequently takes up to 1.2 seconds, while the 400 ms for ext3 is pretty constant.

Right now, nfsd_sync calls

	filemap_fdatawrite
	filp->f_op->fsync
	filemap_fdatawait

all under the i_sem. However, it seems we don't need the i_sem for the filemap_* functions (is that valid - at least sync_page_range doesn't?). So I changed the code to make it grab i_sem only for the fsync call, but unfortunately, that doesn't seem to make much of a difference, as I found out. Most of the time taken by a commit is spent in fsync (the delta between the fsync latency and the overall commit latency is usually less than 5 ms, i.e. ~1%).

I also changed nfsd_sync to call filemap_fdatawrite_range instead of filemap_fdatawrite, but that doesn't make a noticeable difference either.

I then re-enabled my flushfast hack, and the commit latencies went down to 30 ms on ext3, with the occasional spike of 300 ms. On reiser, the commit latency went down to something like 50 ms on average.

(The reiserfs rewrite case was fairly bad, however. Rewrite over NFS on top of reiser is fairly slow to begin with, much slower than write; and the gain from the flushfast patch is minimal - but that's a different story)

Conclusion:

So this at least supports my theory that the commits are throttling the writes quite a bit.

For the sake of completeness, I did some more iozone measurements, and on write/rewrite the performance gain is about 50% on both reiser and ext3, for a single client. I would think for several clients writing concurrently, the gain should be even more pronounced, but I haven't run these tests yet.

I'm wondering what could happen if we change nfsd_sync to not take the i_sem at all... I'll talk to a few VFS folks around here and try to find out.

PS:

Another thing I noticed was that the commit calls sent by the Linux client (2.6.5) are not evenly distributed over time. Much of the time, the client will call COMMIT 4-6 times a second, and then all of a sudden I see 30-80 calls a second several times in a row.

With IRIX clients, which do far fewer COMMITs than (at least 2.4) Linux clients, I have seen COMMIT latencies in the order of multiple seconds as over a gigabyte of data is written to disk at 180 MB/s.

I imagine they will not be thrilled by the idea.

I still think the best approach is to get the page cache to start pushing unstable NFS pages to disk more aggresively, after the WRITE but before the COMMIT. This should avoid long waits for disk IO with i_sem held; IIRC the page cache will only hold i_sem long enough to traverse page lists, allowing another WRITE call to get in soon.

Hi,

I've been looking at the write throughput with NFSv3, and played around a little. Here's a patch that seems to increase iozone's write throughput from 6MB/s to close to 10MB/s on my test machine.

(The improvement in rewrite throughput is less pronounced). I'm still doing some testing on this, but I would still appreciate some feedback, especially from people seeing throughput problems.

Olaf

Signature · 5 lines

Olaf Kirch     |  The Hardware Gods hate me.
[email protected]   |
---------------+ 
Index: linux-2.6.5/fs/nfsd/vfs.c
===================================================================

fs/nfsd/vfs.c +10 -0

@@ -45,6 +45,7 @@
 #include <linux/quotaops.h>
 #include <linux/dnotify.h>
 #include <linux/xattr_acl.h>
+#include <linux/backing-dev.h>
 
 #include <asm/uaccess.h>
 

@@ -810,6 +811,22 @@
 		}
 		last_ino = inode->i_ino;
 		last_dev = inode->i_sb->s_dev;
+	} else if (err >= 0 && !stable) {
+		/* If we've been writing several pages, schedule them
+		 * for the disk immediately. The client may be streaming
+		 * and we don't want to hang on a huge journal sync when the
+		 * commit comes in
+		 */
+		struct address_space	*mapping;
+
+		/* This assumes a minimum page size of 1K, and will issue