Intruduce nfsrahead
Subsystems (2)
v4 cover Intruduce nfsrahead
Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.
Specifically, the read troughput on a sec=krb5p mount drops by 50-75% when comparing the default readahead with a readahead of 15360.
Previous discussions:
https://lore.kernel.org/linux-nfs/[email protected]/
I attempted to add a non-kernel option to mount.nfs, and it was rejected.
https://lore.kernel.org/linux-nfs/[email protected]/
Attempted to add a mount option to the kernel, rejected as well.
I had started a separate tool to set the readahead of BDIs, but the scope is specifically for NFS, so I would like to get the community feeling for having this in nfs-utils.
This patch series introduces nfs-readahead-udev, a utility to automatically set NFS readahead when NFS is mounted. The utility is triggered by udev when a new BDI is added, returns to udev the value of the readahead that should be used.
The tool currently supports setting read ahead per mountpoint, nfs major version, or by a global default value.
v2:
- explain the motivation
v3:
- adopt already available facilities
- nfsrahead is now configured in nfs.conf
v4:
- retry getting the device if it fails
- assorted fixes and improvements
Thiago Becker (7):
Create nfsrahead
nfsrahead: configure udev
nfsrahead: only set readahead for nfs devices.
nfsrahead: add logging
nfsrahead: get the information from the config file.
nfsrahead: User documentation
nfsrahead: retry getting the device if it fails.
.gitignore | 2 + configure.ac | 1 + systemd/nfs.conf.man | 11 ++ tools/Makefile.am | 2 +- tools/nfsrahead/99-nfs.rules.in | 1 + tools/nfsrahead/Makefile.am | 16 +++ tools/nfsrahead/main.c | 183 ++++++++++++++++++++++++++++++++ tools/nfsrahead/nfsrahead.man | 72 +++++++++++++ 8 files changed, 287 insertions(+), 1 deletion(-) create mode 100644 tools/nfsrahead/99-nfs.rules.in create mode 100644 tools/nfsrahead/Makefile.am create mode 100644 tools/nfsrahead/main.c create mode 100644 tools/nfsrahead/nfsrahead.man
Signature · 1 line
2.35.1
PATCH v4 1/7 Create nfsrahead
Kernel commit c128e575514c ("NFS: Optimise the default readahead size") changed the calculation for NFS readahead from a multiple of rsize to the system default, 128 kiB. This setting has been causing read heavy workloads to underperform by at least 30%, as show below.
$ cat /sys/class/bdi/0\:55/read_ahead_kb 128
$ for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 \
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.056 s, 252 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.1258 s, 251 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5981 s, 259 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5487 s, 260 MB/s$ echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb
$ for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 \
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.3855 s, 347 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2528 s, 382 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.9849 s, 358 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2953 s, 380 MB/sThis patch and the following create a tool for automatically setting NFS readahead during the mount, nfsrahead. The tool is invoked from udev when the NFS backing device is created in kernel, and sets the readahead using the sysfs interface.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- .gitignore | 1 + configure.ac | 1 + tools/Makefile.am | 2 +- tools/nfsrahead/Makefile.am | 3 +++ tools/nfsrahead/main.c | 7 +++++++ 5 files changed, 13 insertions(+), 1 deletion(-) create mode 100644 tools/nfsrahead/Makefile.am create mode 100644 tools/nfsrahead/main.c
.gitignore +1 -0
@@ -61,6 +61,7 @@ utils/statd/statd tools/locktest/testlk tools/getiversion/getiversion tools/nfsconf/nfsconf +tools/nfsrahead/nfsrahead support/export/mount.h support/export/mount_clnt.c support/export/mount_xdr.c
configure.ac +1 -0
@@ -737,6 +737,7 @@ AC_CONFIG_FILES([ tools/rpcgen/Makefile tools/mountstats/Makefile tools/nfs-iostat/Makefile + tools/nfsrahead/Makefile tools/rpcctl/Makefile tools/nfsdclnts/Makefile tools/nfsconf/Makefile
tools/Makefile.am +1 -1
@@ -12,6 +12,6 @@ if CONFIG_NFSDCLD OPTDIRS += nfsdclddb endif -SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts $(OPTDIRS) +SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts nfsrahead $(OPTDIRS) MAINTAINERCLEANFILES = Makefile.in
tools/nfsrahead/Makefile.am new file +3 -0
@@ -0,0 +1,3 @@ +libexec_PROGRAMS = nfsrahead +nfsrahead_SOURCES = main.c +
tools/nfsrahead/main.c new file +7 -0
@@ -0,0 +1,7 @@ +#include <stdio.h> + +int main (int argc,char **argv) +{ +unsigned int readahead =128 ; +printf ("%d\n" , readahead); +}
Signature · 1 line
2.35.1
PATCH v4 2/7 nfsrahead: configure udev
In reply to Thiago BeckerSet the udev rule to call the readahead utility.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- .gitignore | 1 + tools/nfsrahead/99-nfs.rules.in | 1 + tools/nfsrahead/Makefile.am | 8 ++++++++ 3 files changed, 10 insertions(+) create mode 100644 tools/nfsrahead/99-nfs.rules.in
.gitignore +1 -0
@@ -62,6 +62,7 @@ tools/locktest/testlk tools/getiversion/getiversion tools/nfsconf/nfsconf tools/nfsrahead/nfsrahead +tools/nfsrahead/99-nfs_bdi.rules support/export/mount.h support/export/mount_clnt.c support/export/mount_xdr.c
tools/nfsrahead/99-nfs.rules.in new file +1 -0
@@ -0,0 +1 @@ +SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead", ATTR{read_ahead_kb}="%c"
tools/nfsrahead/Makefile.am +8 -0
@@ -1,3 +1,11 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c +udev_rulesdir = /usr/lib/udev/rules.d/ +udev_rules_DATA = 99-nfs.rules + +99-nfs.rules: 99-nfs.rules.in $(builddefs) + $(SED) "s|_libexecdir_|@libexecdir@|g" 99-nfs.rules.in > $@ + +clean-local: + $(RM) 99-nfs.rules
Signature · 1 line
2.35.1
PATCH v4 3/7 nfsrahead: only set readahead for nfs devices.
In reply to Thiago BeckerLimit setting the readahead for nfs devices.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/99-nfs.rules.in | 2 +- tools/nfsrahead/Makefile.am | 1 + tools/nfsrahead/main.c | 128 ++++++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 1 deletion(-)
tools/nfsrahead/99-nfs.rules.in +1 -1
@@ -1 +1 @@ -SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead", ATTR{read_ahead_kb}="%c" +SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead %k", ATTR{read_ahead_kb}="%c"
tools/nfsrahead/Makefile.am +1 -0
@@ -1,5 +1,6 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c +nfsrahead_LDFLAGS= -lmount udev_rulesdir = /usr/lib/udev/rules.d/ udev_rules_DATA = 99-nfs.rules
tools/nfsrahead/main.c +128 -0
135 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v4 4/7 nfsrahead: add logging
In reply to Thiago BeckerBugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/Makefile.am | 1 + tools/nfsrahead/main.c | 40 ++++++++++++++++++++++++++++++++----- 2 files changed, 36 insertions(+), 5 deletions(-)
tools/nfsrahead/Makefile.am +1 -0
@@ -1,6 +1,7 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c nfsrahead_LDFLAGS= -lmount +nfsrahead_LDADD = ../../support/nfs/libnfsconf.la udev_rulesdir = /usr/lib/udev/rules.d/ udev_rules_DATA = 99-nfs.rules
tools/nfsrahead/main.c +35 -5
@@ -2,14 +2,19 @@#include <string.h> #include <stdlib.h> #include <errno.h> +#include <unistd.h> #include <libmount/libmount.h> #include <sys/sysmacros.h> +#include "xlog.h" +#ifndef MOUNTINFO_PATH #define MOUNTINFO_PATH "/proc/self/mountinfo"#endif +#define CONF_NAME "nfsrahead" +/* Device information from the system */ struct device_info {char *device_number;
@@ -108,25 +113,50 @@ static int get_device_info(const char *device_number, struct device_info *devicereturn ret; } +#define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL) +int main (int argc,char **argv) {int ret =0 ;struct device_info device; -unsigned int readahead =128 ; - -if (argc !=2 ) { -return -EINVAL ; +unsigned int readahead =128 , verbose =0 , log_stderr =0 ; +char opt; + +while ((opt =getopt (argc, argv,"dF" )) !=-1 ) { +switch (opt) { +case 'd' : + verbose =1 ; +break ; +case 'F' : + log_stderr =1 ; +break ; + } } -if ((ret =get_device_info (argv[1 ], &device)) !=0 ) { +xlog_stderr (log_stderr); +xlog_syslog (~log_stderr); +xlog_config (L_DEFAULT | (L_NOTICE & verbose),1 ); +xlog_open (CONF_NAME ); + +// xlog_err causes the system to exit +if ((argc - optind) !=1 ) +xlog_err ("expected the device number of a BDI; is udev ok?" ); + +if ((ret =get_device_info (argv[optind], &device)) !=0 ) { +xlog (L_ERROR ,"unable to find device %s\n" , argv[optind]); goto out; }if (strncmp ("nfs" , device.fstype,3 ) !=0 ) { +xlog (L_NOTICE , +"not setting readahead for non supported fstype %s on device %s\n" , + device.fstype, argv[optind]); ret = -EINVAL ; goto out; } +xlog (L_WARNING ,"setting %s readahead to %d\n" , device.mountpoint, readahead); +printf ("%d\n" , readahead); out:
Signature · 1 line
2.35.1
PATCH v4 5/7 nfsrahead: get the information from the config file.
In reply to Thiago BeckerBugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/main.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/nfsrahead/main.c +14 -0
@@ -8,12 +8,14 @@#include <sys/sysmacros.h> #include "xlog.h" +#include "conffile.h" #ifndef MOUNTINFO_PATH #define MOUNTINFO_PATH "/proc/self/mountinfo"#endif #define CONF_NAME "nfsrahead" +#define NFS_DEFAULT_READAHEAD 128/* Device information from the system */ struct device_info {
@@ -113,6 +115,14 @@ static int get_device_info(const char *device_number, struct device_info *devicereturn ret; } +static int conf_get_readahead (const char *kind) { +int readahead =0 ; + +if ((readahead =conf_get_num (CONF_NAME , kind,-1 )) ==-1 ) + readahead =conf_get_num (CONF_NAME ,"default" ,NFS_DEFAULT_READAHEAD ); + +return readahead; +}#define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL)int main (int argc,char **argv)
@@ -133,6 +143,8 @@ int main(int argc, char **argv) } } +conf_init_file (NFS_CONFFILE ); +xlog_stderr (log_stderr);xlog_syslog (~log_stderr);xlog_config (L_DEFAULT | (L_NOTICE & verbose),1 );
@@ -155,6 +167,8 @@ int main(int argc, char **argv) goto out; } + readahead =conf_get_readahead (device.fstype); +xlog (L_WARNING ,"setting %s readahead to %d\n" , device.mountpoint, readahead);printf ("%d\n" , readahead);
Signature · 1 line
2.35.1
PATCH v4 6/7 nfsrahead: User documentation
In reply to Thiago BeckerAdd the man page for nfsrahead, and add the new section to nfs.conf.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- systemd/nfs.conf.man | 11 ++++++ tools/nfsrahead/Makefile.am | 3 ++ tools/nfsrahead/nfsrahead.man | 72 +++++++++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+) create mode 100644 tools/nfsrahead/nfsrahead.man
systemd/nfs.conf.man +11 -0
@@ -294,6 +294,17 @@ Only .B debug= is recognized. +.TP +.B nfsrahead +Recognized values: +.BR nfs , +.BR nfsv4 , +.BR default . + +See +.BR nfsrahead (5) +for deatils. + .SH FILES .TP 10n .I /etc/nfs.conf
tools/nfsrahead/Makefile.am +3 -0
@@ -3,6 +3,9 @@ nfsrahead_SOURCES = main.c nfsrahead_LDFLAGS= -lmount nfsrahead_LDADD = ../../support/nfs/libnfsconf.la +man5_MANS = nfsrahead.man +EXTRA_DIST = $(man5_MANS) + udev_rulesdir = /usr/lib/udev/rules.d/ udev_rules_DATA = 99-nfs.rules
tools/nfsrahead/nfsrahead.man new file +72 -0
72 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v4 7/7 nfsrahead: retry getting the device if it fails.
In reply to Thiago BeckerSometimes the mountinfo entry is not available when nfsrahead is called, leading to failure to set the readahead. Retry getting the device before failing.
--- tools/nfsrahead/main.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tools/nfsrahead/main.c +6 -2
@@ -127,7 +127,7 @@ static int conf_get_readahead(const char *kind) {int main (int argc,char **argv) { -int ret =0 ; +int ret =0 , retry;struct device_info device;unsigned int readahead =128 , verbose =0 , log_stderr =0 ;char opt;
@@ -154,7 +154,11 @@ int main(int argc, char **argv)if ((argc - optind) !=1 )xlog_err ("expected the device number of a BDI; is udev ok?" ); -if ((ret =get_device_info (argv[optind], &device)) !=0 ) { +for (retry =0 ; retry <=10 ; retry++ ) +if ((ret =get_device_info (argv[optind], &device)) ==0 ) +break ; + +if (ret !=0 ) {xlog (L_ERROR ,"unable to find device %s\n" , argv[optind]); goto out; }
Signature · 1 line
2.35.1
Quoted 57 lines · Thiago Becker
Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.
Specifically, the read troughput on a sec=krb5p mount drops by 50-75% when comparing the default readahead with a readahead of 15360.
Previous discussions:
https://lore.kernel.org/linux-nfs/[email protected]/
I attempted to add a non-kernel option to mount.nfs, and it was rejected.https://lore.kernel.org/linux-nfs/[email protected]/
Attempted to add a mount option to the kernel, rejected as well.I had started a separate tool to set the readahead of BDIs, but the scope is specifically for NFS, so I would like to get the community feeling for having this in nfs-utils.
This patch series introduces nfs-readahead-udev, a utility to automatically set NFS readahead when NFS is mounted. The utility is triggered by udev when a new BDI is added, returns to udev the value of the readahead that should be used.
The tool currently supports setting read ahead per mountpoint, nfs major version, or by a global default value.
v2:
- explain the motivationv3:
- adopt already available facilities
- nfsrahead is now configured in nfs.confv4:
- retry getting the device if it fails
- assorted fixes and improvementsThiago Becker (7):
Create nfsrahead
nfsrahead: configure udev
nfsrahead: only set readahead for nfs devices.
nfsrahead: add logging
nfsrahead: get the information from the config file.
nfsrahead: User documentation
nfsrahead: retry getting the device if it fails..gitignore | 2 + configure.ac | 1 + systemd/nfs.conf.man | 11 ++ tools/Makefile.am | 2 +- tools/nfsrahead/99-nfs.rules.in | 1 + tools/nfsrahead/Makefile.am | 16 +++ tools/nfsrahead/main.c | 183 ++++++++++++++++++++++++++++++++ tools/nfsrahead/nfsrahead.man | 72 +++++++++++++ 8 files changed, 287 insertions(+), 1 deletion(-)
Quoted 5 lines
create mode 100644
tools/nfsrahead/99-nfs.rules.increate mode 100644tools/nfsrahead/Makefile.amcreate mode 100644tools/nfsrahead/main.ccreate mode 100644tools/nfsrahead/nfsrahead.man
Committed... (tag nfs-utils-2-6-2-rc4)
steved.