Skip to contentSkip to conversation

Make NFS-Client readahead tunable

RFC In review Martin Knoblauch 6 participants 6 messages Sep 17, 2008 – Sep 18, 2008 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 Make NFS-Client readahead tunable

Hi,

the following/attached patch works around a [obscure] problem when an 2.6 (not sure/caring about 2.4) NFS client accesses an "offline" file on a Sun/Solaris-10 NFS server when the underlying filesystem is of type SAM-FS. Happens with RHEL4/5 and mainline kernels. Frankly, it is not a Linux problem, but the chance for a short-/mid-term solution from Sun are very slim. So, being lazy, I would love to get this patch into Linux. If not, I just will have to maintain it for eternity out of tree.

The problem: SAM-FS is Suns proprietary HSM filesystem. It stores meta-data and a relatively small amount of data "online" on disk and pushes old or infrequently used data to "offline" media like e.g. tape. This is completely transparent to the users. If the date for an "offline" file is needed, the so called "stager daemon" copies it back from the offline medium. All of this works great most of the time. Now, if an Linux NFS client tries to read such an offline file, performance drops to "extremely slow". After lengthly investigation of tcp-dumps, mount options and procedures involving black cats at midnight, we found out that the readahead behaviour of the Linux NFS client causes the problem. Basically it seems to issue read requests up to 15*rsize to the server. In the case of the "offl
ine" files, this behaviour causes heavy competition for the inode lock between the NFSD process and the stager daemon on the Solaris server.

- The real solution: fixing SAM-FS/NFSD interaction. Sun engineering acks the problem, but a solution will need time. Lots of it.
- The working solution: disable the client side readahead, or make it tunable. The patch does that by introducing a NFS module parameter "ra_factor" which can take values between 1 and 15 (default 15) and a tunable "/proc/sys/fs/nfs/nfs_ra_factor" with the same range and default.

fs/nfs/client.c 2008-09-17 11:55:18.000000000 +0200 renamed from fs/nfs/client.c 2008-09-17 11:35:21.000000000 +0200 +10 -1
diff -urp linux-2.6.27-rc6-git4/fs/nfs/client.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/client.c
@@ -722,6 +722,11 @@ error:
 }
 
 /*
+ * NFS Client Read-Ahead factor
+*/
+unsigned int nfs_ra_factor;
+
+/*
  * Load up the server record from information gained in an fsinfo record
  */
 static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *fsinfo)
@@ -746,7 +751,11 @@ static void nfs_server_set_fsinfo(struct
                server->rsize = NFS_MAX_FILE_IO_SIZE;
        server->rpages = (server->rsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
 
-       server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
+       dprintk("nfs_server_set_fsinfo: rsize, wsize, rpages, \
+               nfs_ra_factor, ra_pages: %d %d %d %d %d\n",
+               server->rsize,server->wsize,server->rpages,
+               nfs_ra_factor,server->rpages * nfs_ra_factor);
+       server->backing_dev_info.ra_pages = server->rpages * nfs_ra_factor;
 
        if (server->wsize > max_rpc_payload)
                server->wsize = max_rpc_payload;
fs/nfs/inode.c 2008-09-17 11:45:09.000000000 +0200 renamed from fs/nfs/inode.c 2008-09-17 11:35:21.000000000 +0200 +12 -0
diff -urp linux-2.6.27-rc6-git4/fs/nfs/inode.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/inode.c
@@ -53,6 +53,8 @@
 
 /* Default is to see 64-bit inode numbers */
 static int enable_ino64 = NFS_64_BIT_INODE_NUMBERS_ENABLED;
+static unsigned int ra_factor __read_mostly = NFS_MAX_READAHEAD;
+
 
 static void nfs_invalidate_inode(struct inode *);
 static int nfs_update_inode(struct inode *, struct nfs_fattr *);
@@ -1347,6 +1349,12 @@ static int __init init_nfs_fs(void)
 #endif
        if ((err = register_nfs_fs()) != 0)
                goto out;
+
+       if (ra_factor < 1 || ra_factor > NFS_MAX_READAHEAD)
+               nfs_ra_factor = NFS_MAX_READAHEAD;
+       else
+               nfs_ra_factor = ra_factor;
+
        return 0;
 out:
 #ifdef CONFIG_PROC_FS
@@ -1388,6 +1396,10 @@ static void __exit exit_nfs_fs(void)
 MODULE_AUTHOR("Olaf Kirch <[email protected]>");
 MODULE_LICENSE("GPL");
 module_param(enable_ino64, bool, 0644);
+MODULE_PARM_DESC(enable_ino64, "Enable 64-bit inode numbers (Default: 1)");
+module_param(ra_factor, uint, 0644);
+MODULE_PARM_DESC(ra_factor,
+       "Number of rsize read-ahead requests (Default/Max: 15, Min: 1)");
 
 module_init(init_nfs_fs)
 module_exit(exit_nfs_fs)
fs/nfs/sysctl.c 2008-09-17 11:45:09.000000000 +0200 renamed from fs/nfs/sysctl.c 2008-07-13 23:51:29.000000000 +0200 +13 -0
diff -urp linux-2.6.27-rc6-git4/fs/nfs/sysctl.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/sysctl.c
@@ -14,9 +14,12 @@
 #include <linux/nfs_fs.h>
 
 #include "callback.h"
+#include "internal.h"
 
 static const int nfs_set_port_min = 0;
 static const int nfs_set_port_max = 65535;
+static const unsigned int min_nfs_ra_factor = 1;
+static const unsigned int max_nfs_ra_factor = NFS_MAX_READAHEAD;
 static struct ctl_table_header *nfs_callback_sysctl_table;
 
 static ctl_table nfs_cb_sysctls[] = {
@@ -58,6 +61,16 @@ static ctl_table nfs_cb_sysctls[] = {
                .mode           = 0644,
                .proc_handler   = &proc_dointvec,
        },
+       {
+               .ctl_name = CTL_UNNUMBERED,
+               .procname = "nfs_ra_factor",
+               .data = &nfs_ra_factor,
+               .maxlen = sizeof(unsigned int),
+               .mode = 0644,
+               .proc_handler = &proc_dointvec_minmax,
+               .extra1 = (unsigned int *)&min_nfs_ra_factor,
+               .extra2 = (unsigned int *)&max_nfs_ra_factor,
+       },
        { .ctl_name = 0 }
 };
 
include/linux/nfs_fs.h 2008-09-17 11:45:09.000000000 +0200 renamed from include/linux/nfs_fs.h 2008-09-17 11:35:25.000000000 +0200 +5 -0
diff -urp linux-2.6.27-rc6-git4/include/linux/nfs_fs.h linux-2.6.27-rc6-git4-nfs_ra/include/linux/nfs_fs.h
@@ -464,6 +464,11 @@ extern int nfs_writeback_done(struct rpc
 extern void nfs_writedata_release(void *);
 
 /*
+ * linux/fs/nfs/client.c
+*/
+extern unsigned int nfs_ra_factor;
+
+/*
  * Try to write back everything synchronously (but check the
  * return value!)
  */
Makefile 2008-09-17 11:45:09.000000000 +0200 renamed from Makefile 2008-09-17 11:35:56.000000000 +0200 +1 -1
diff -urp linux-2.6.27-rc6-git4/Makefile linux-2.6.27-rc6-git4-nfs_ra/Makefile
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 27
-EXTRAVERSION = -rc6-git4
+EXTRAVERSION = -rc6-git4-nfs_ra
 NAME = Rotary Wombat
 
 # *DOCUMENTATION*

Cheers
Martin

------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de

fs/nfs/client.c +10 -1
diff -urp linux-2.6.27-rc6-git4/fs/nfs/client.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/client.c
@@ -722,6 +722,11 @@ error:
 }
 
 /*
+ * NFS Client Read-Ahead factor
+*/
+unsigned int nfs_ra_factor;
+
+/*
  * Load up the server record from information gained in an fsinfo record
  */
 static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *fsinfo)
@@ -746,7 +751,11 @@ static void nfs_server_set_fsinfo(struct
 		server->rsize = NFS_MAX_FILE_IO_SIZE;
 	server->rpages = (server->rsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
 
-	server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
+	dprintk("nfs_server_set_fsinfo: rsize, wsize, rpages, \
+		nfs_ra_factor, ra_pages: %d %d %d %d %d\n",
+		server->rsize,server->wsize,server->rpages,
+		nfs_ra_factor,server->rpages * nfs_ra_factor);
+	server->backing_dev_info.ra_pages = server->rpages * nfs_ra_factor;
 
 	if (server->wsize > max_rpc_payload)
 		server->wsize = max_rpc_payload;
fs/nfs/inode.c +12 -0
diff -urp linux-2.6.27-rc6-git4/fs/nfs/inode.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/inode.c
@@ -53,6 +53,8 @@
 
 /* Default is to see 64-bit inode numbers */
 static int enable_ino64 = NFS_64_BIT_INODE_NUMBERS_ENABLED;
+static unsigned int ra_factor __read_mostly = NFS_MAX_READAHEAD;
+
 
 static void nfs_invalidate_inode(struct inode *);
 static int nfs_update_inode(struct inode *, struct nfs_fattr *);
@@ -1347,6 +1349,12 @@ static int __init init_nfs_fs(void)
 #endif
 	if ((err = register_nfs_fs()) != 0)
 		goto out;
+
+	if (ra_factor < 1 || ra_factor > NFS_MAX_READAHEAD)
+		nfs_ra_factor = NFS_MAX_READAHEAD;
+	else
+		nfs_ra_factor = ra_factor;
+
 	return 0;
 out:
 #ifdef CONFIG_PROC_FS
@@ -1388,6 +1396,10 @@ static void __exit exit_nfs_fs(void)
 MODULE_AUTHOR("Olaf Kirch <[email protected]>");
 MODULE_LICENSE("GPL");
 module_param(enable_ino64, bool, 0644);
+MODULE_PARM_DESC(enable_ino64, "Enable 64-bit inode numbers (Default: 1)");
+module_param(ra_factor, uint, 0644);
+MODULE_PARM_DESC(ra_factor,
+	"Number of rsize read-ahead requests (Default/Max: 15, Min: 1)");
 
 module_init(init_nfs_fs)
 module_exit(exit_nfs_fs)
fs/nfs/sysctl.c +13 -0
diff -urp linux-2.6.27-rc6-git4/fs/nfs/sysctl.c linux-2.6.27-rc6-git4-nfs_ra/fs/nfs/sysctl.c
@@ -14,9 +14,12 @@
 #include <linux/nfs_fs.h>
 
 #include "callback.h"
+#include "internal.h"
 
 static const int nfs_set_port_min = 0;
 static const int nfs_set_port_max = 65535;
+static const unsigned int min_nfs_ra_factor = 1;
+static const unsigned int max_nfs_ra_factor = NFS_MAX_READAHEAD;
 static struct ctl_table_header *nfs_callback_sysctl_table;
 
 static ctl_table nfs_cb_sysctls[] = {
@@ -58,6 +61,16 @@ static ctl_table nfs_cb_sysctls[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
+	{
+		.ctl_name = CTL_UNNUMBERED,
+		.procname = "nfs_ra_factor",
+		.data = &nfs_ra_factor,
+		.maxlen = sizeof(unsigned int),
+		.mode = 0644,
+		.proc_handler = &proc_dointvec_minmax,
+		.extra1 = (unsigned int *)&min_nfs_ra_factor,
+		.extra2 = (unsigned int *)&max_nfs_ra_factor,
+	},
 	{ .ctl_name = 0 }
 };
 
include/linux/nfs_fs.h +5 -0
diff -urp linux-2.6.27-rc6-git4/include/linux/nfs_fs.h linux-2.6.27-rc6-git4-nfs_ra/include/linux/nfs_fs.h
@@ -464,6 +464,11 @@ extern int nfs_writeback_done(struct rpc
 extern void nfs_writedata_release(void *);
 
 /*
+ * linux/fs/nfs/client.c
+*/
+extern unsigned int nfs_ra_factor;
+
+/*
  * Try to write back everything synchronously (but check the
  * return value!)
  */
Makefile +1 -1
diff -urp linux-2.6.27-rc6-git4/Makefile linux-2.6.27-rc6-git4-nfs_ra/Makefile
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 27
-EXTRAVERSION = -rc6-git4
+EXTRAVERSION = -rc6-git4-nfs_ra
 NAME = Rotary Wombat
 
 # *DOCUMENTATION*

A message that is not in the archive (sent off-list or not yet received): 997439.5560.qm-VAEUvbQToQWvuULXzWHTWIglqE1Y4D90QQ4Iyu8u01E@public.gmane.org

#2 lore raw

There are times when you want to increase nfs readahead instead of decreasing it. I suggest you make the default 15 but allow tuning it either up or down.

#3 lore raw
In reply to Martin Knoblauch
Quoted 5 lines · Martin Knoblauch

Hi,

the following/attached patch works around a [obscure] problem when an 2.6 (not sure/caring about 2.4) NFS client accesses an "offline" file on a Sun/Solaris-10 NFS server when the underlying filesystem is of type SAM-FS. Happens with RHEL4/5 and mainline kernels. Frankly, it is not a Linux problem, but the chance for a short-/mid-term solution from Sun are very slim. So, being lazy, I would love to get this patch into Linux. If not, I just will have to maintain it for eternity out of tree.

The problem: SAM-FS is Suns proprietary HSM filesystem. It stores meta-data and a relatively small amount of data "online" on disk and pushes old or infrequently used data to "offline" media like e.g. tape. This is completely transparent to the users. If the date for an "offline" file is needed, the so called "stager daemon" copies it back from the offline medium. All of this works great most of the time. Now, if an Linux NFS client tries to read such an offline file, performance drops to "extremely slow". After lengthly investigation of tcp-dumps, mount options and procedures involving black cats at midnight, we found out that the readahead behaviour of the Linux NFS client causes the problem. Basically it seems to issue read requests up to 15*rsize to the server. In the case of the "of

fline" files, this behaviour causes heavy competition for the inode lock between the NFSD process and the stager daemon on the Solaris server.

Quoted 3 lines

- The real solution: fixing SAM-FS/NFSD interaction. Sun engineering acks the problem, but a solution will need time. Lots of it.
- The working solution: disable the client side readahead, or make it tunable. The patch does that by introducing a NFS module parameter "ra_factor" which can take values between 1 and 15 (default 15) and a tunable "/proc/sys/fs/nfs/nfs_ra_factor" with the same range and default.

Hi.

I was curious if a design to limit or eliminate read-ahead activity when the server returns EJUKEBOX was considered? Unless one can know that the server and client can get into this situation ahead of time, how would the tunable be used?

Thanx...

ps

#4 lore raw
Quoted 33 lines · Peter Staubach
Quoted 25 lines · Martin Knoblauch

Hi,

the following/attached patch works around a [obscure] problem when an 2.6 (not sure/caring about 2.4) NFS client accesses an "offline" file on a Sun/Solaris-10 NFS server when the underlying filesystem is of type SAM-FS. Happens with RHEL4/5 and mainline kernels. Frankly, it is not a Linux problem, but the chance for a short-/mid-term solution from Sun are very slim. So, being lazy, I would love to get this patch into Linux. If not, I just will have to maintain it for eternity out of tree.

The problem: SAM-FS is Suns proprietary HSM filesystem. It stores meta-data and a relatively small amount of data "online" on disk and pushes old or infrequently used data to "offline" media like e.g. tape. This is completely transparent to the users. If the date for an "offline" file is needed, the so called "stager daemon" copies it back from the offline medium. All of this works great most of the time. Now, if an Linux NFS client tries to read such an offline file, performance drops to "extremely slow". After lengthly investigation of tcp-dumps, mount options and procedures involving black cats at midnight, we found out that the readahead behaviour of the Linux NFS client causes the problem. Basically it seems to issue read requests up to 15*rsize to the server. In the case of the "offline" files, this behaviour causes heavy competition for the inode lock between the NFSD process and the stager daemon on the Solaris server.

Quoted 6 lines

- The real solution: fixing SAM-FS/NFSD interaction. Sun engineering acks the problem, but a solution will need time. Lots of it.
- The working solution: disable the client side readahead, or make it tunable. The patch does that by introducing a NFS module parameter "ra_factor" which can take values between 1 and 15 (default 15) and a tunable "/proc/sys/fs/nfs/nfs_ra_factor" with the same range and default.

Quoted 6 lines

Hi.

I was curious if a design to limit or eliminate read-ahead activity when the server returns EJUKEBOX was considered? Unless one can know that the server and client can get into this situation ahead of time, how would the tunable be used?

I tend to agree. A tunable is probably not a good solution in this case.

I would bet that this lock contention issue is a problem in other more common cases, and would merit some careful analysis.

Signature · 1 line
Chuck Lever
#5 lore raw
In reply to Martin Knoblauch
Quoted 5 lines · Martin Knoblauch

Hi,

the following/attached patch works around a [obscure] problem when an 2.6 (not sure/caring about 2.4) NFS client accesses an "offline" file on a Sun/Solaris-10 NFS server when the underlying filesystem is of type SAM-FS. Happens with RHEL4/5 and mainline kernels. Frankly, it is not a Linux problem, but the chance for a short-/mid-term solution from Sun are very slim. So, being lazy, I would love to get this patch into Linux. If not, I just will have to maintain it for eternity out of tree.

The problem: SAM-FS is Suns proprietary HSM filesystem. It stores meta-data and a relatively small amount of data "online" on disk and pushes old or infrequently used data to "offline" media like e.g. tape. This is completely transparent to the users. If the date for an "offline" file is needed, the so called "stager daemon" copies it back from the offline medium. All of this works great most of the time. Now, if an Linux NFS client tries to read such an offline file, performance drops to "extremely slow".

By "extremely slow" do you mean "tape read speed"?

Quoted 2 lines

After lengthly investigation of tcp-dumps, mount options and procedures involving black cats at midnight, we found out that the readahead behaviour of the Linux NFS client causes the problem. Basically it seems to issue read requests up to 15*rsize to the server. In the case of the "offline" files, this behaviour causes heavy competition for the inode lock between the NFSD process and the stager daemon on the Solaris server.

So, you need to

a) make your stager daemon do IO more sensibly, and

b) apply something like this patch which adds O_NONBLOCK when knfsd does reads writes and truncates and translates -EAGAIN into NFS3ERR_JUKEBOX

http://kerneltrap.org/mailarchive/linux-fsdevel/2006/5/5/312567

and

c) make your filesystem IO interposing layer report -EAGAIN when a process tries to do IO to an offline region in a file and O_NONBLOCK is present.

Quoted 3 lines

- The real solution: fixing SAM-FS/NFSD interaction. Sun engineering acks the problem, but a solution will need time. Lots of it.
- The working solution: disable the client side readahead, or make it tunable. The patch does that by introducing a NFS module parameter "ra_factor" which can take values between 1 and 15 (default 15) and a tunable "/proc/sys/fs/nfs/nfs_ra_factor" with the same range and default.

I think having a tunable for client readahead is an excellent idea, although not to solve your particular problem. The SLES10 kernel has a patch which does precisely that, perhaps Neil could post it.

I don't think there's a lot of point having both a module parameter and a sysctl.

A maximum of 15 is unwise. I've found that (at least with the older readahead mechanisms in SLES10) a multiple of 4 is required to preserve rsize-alignment of READ rpcs to the server, which helps a lot with wide RAID backends. So in SGI we tune client readahead to 16.

Your patch seems to have a bunch of other unrelated stuff mixed in.

Signature · 3 lines
Greg Banks, P.Engineer, SGI Australian Software Group.
Be like the squirrel.
I don't speak for SGI.
#6 lore raw
Quoted 6 lines · On Thu, 18 Sep 2008 11:42:54 +1000 Greg Banks…

I think having a tunable for client readahead is an excellent idea, although not to solve your particular problem. The SLES10 kernel has a patch which does precisely that, perhaps Neil could post it.

I don't think there's a lot of point having both a module parameter and a sysctl.

mount -o remount,readahead=42