Intruduce nfsrahead
Subsystems (4)
v3 cover Intruduce nfsrahead
Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.
Specifically, the read troughput on a sec=krb5p mount drops by 50-75% when comparing the default readahead with a readahead of 15360.
Previous discussions:
https://lore.kernel.org/linux-nfs/[email protected]/
I attempted to add a non-kernel option to mount.nfs, and it was rejected.
https://lore.kernel.org/linux-nfs/[email protected]/
Attempted to add a mount option to the kernel, rejected as well.
I had started a separate tool to set the readahead of BDIs, but the scope is specifically for NFS, so I would like to get the community feeling for having this in nfs-utils.
This patch series introduces nfs-readahead-udev, a utility to automatically set NFS readahead when NFS is mounted. The utility is triggered by udev when a new BDI is added, returns to udev the value of the readahead that should be used.
The tool currently supports setting read ahead per mountpoint, nfs major version, or by a global default value.
v2:
- explain the motivation
v3:
- adopt already available facilities
- nfsrahead is now configured in nfs.conf
Thiago Becker (6):
Create nfsrahead
nfsrahead: configure udev
nfsrahead: only set readahead for nfs devices.
nfsrahead: add logging
hfsrahead: get the information from the config file.
nfsrahead: User documentation
.gitignore | 2 + configure.ac | 1 + systemd/nfs.conf.man | 11 ++ tools/Makefile.am | 2 +- tools/nfsrahead/99-nfs_bdi.rules.in | 1 + tools/nfsrahead/Makefile.am | 15 +++ tools/nfsrahead/main.c | 179 ++++++++++++++++++++++++++++ tools/nfsrahead/nfsrahead.man | 72 +++++++++++ 8 files changed, 282 insertions(+), 1 deletion(-) create mode 100644 tools/nfsrahead/99-nfs_bdi.rules.in create mode 100644 tools/nfsrahead/Makefile.am create mode 100644 tools/nfsrahead/main.c create mode 100644 tools/nfsrahead/nfsrahead.man
Signature · 1 line
2.35.1
RFC PATCH v3 1/6 Create nfsrahead
This tool is invoked by udev to find and set the readahead value to NFS mounts.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- .gitignore | 1 + configure.ac | 1 + tools/Makefile.am | 2 +- tools/nfsrahead/Makefile.am | 3 +++ tools/nfsrahead/main.c | 7 +++++++ 5 files changed, 13 insertions(+), 1 deletion(-) create mode 100644 tools/nfsrahead/Makefile.am create mode 100644 tools/nfsrahead/main.c
.gitignore +1 -0
@@ -61,6 +61,7 @@ utils/statd/statd tools/locktest/testlk tools/getiversion/getiversion tools/nfsconf/nfsconf +tools/nfsrahead/nfsrahead support/export/mount.h support/export/mount_clnt.c support/export/mount_xdr.c
configure.ac +1 -0
@@ -737,6 +737,7 @@ AC_CONFIG_FILES([ tools/rpcgen/Makefile tools/mountstats/Makefile tools/nfs-iostat/Makefile + tools/nfsrahead/Makefile tools/rpcctl/Makefile tools/nfsdclnts/Makefile tools/nfsconf/Makefile
tools/Makefile.am +1 -1
@@ -12,6 +12,6 @@ if CONFIG_NFSDCLD OPTDIRS += nfsdclddb endif -SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts $(OPTDIRS) +SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts nfsrahead $(OPTDIRS) MAINTAINERCLEANFILES = Makefile.in
tools/nfsrahead/Makefile.am new file +3 -0
@@ -0,0 +1,3 @@ +libexec_PROGRAMS = nfsrahead +nfsrahead_SOURCES = main.c +
tools/nfsrahead/main.c new file +7 -0
@@ -0,0 +1,7 @@ +#include <stdio.h> + +int main (int argc,char **argv,char **envp) +{ +unsigned int readahead =128 ; +printf ("%d\n" , readahead); +}
Signature · 1 line
2.35.1
RFC PATCH v3 2/6 nfsrahead: configure udev
In reply to Thiago BeckerSet the udev rule to call the readahead utility.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- .gitignore | 1 + tools/nfsrahead/99-nfs_bdi.rules.in | 1 + tools/nfsrahead/Makefile.am | 8 ++++++++ 3 files changed, 10 insertions(+) create mode 100644 tools/nfsrahead/99-nfs_bdi.rules.in
.gitignore +1 -0
@@ -62,6 +62,7 @@ tools/locktest/testlk tools/getiversion/getiversion tools/nfsconf/nfsconf tools/nfsrahead/nfsrahead +tools/nfsrahead/99-nfs_bdi.rules support/export/mount.h support/export/mount_clnt.c support/export/mount_xdr.c
tools/nfsrahead/99-nfs_bdi.rules.in new file +1 -0
@@ -0,0 +1 @@ +SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead", ATTR{read_ahead_kb}="%c"
tools/nfsrahead/Makefile.am +8 -0
@@ -1,3 +1,11 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c +udev_rulesdir = /etc/udev/rules.d +udev_rules_DATA = 99-nfs_bdi.rules + +99-nfs_bdi.rules: 99-nfs_bdi.rules.in $(builddefs) + $(SED) "s|_libexecdir_|@libexecdir@|g" 99-nfs_bdi.rules.in > $@ + +clean-local: + $(RM) 99-nfs_bdi.rules
Signature · 1 line
2.35.1
RFC PATCH v3 3/6 nfsrahead: only set readahead for nfs devices.
In reply to Thiago BeckerLimit setting the readahead for nfs devices.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/Makefile.am | 1 + tools/nfsrahead/main.c | 130 ++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+)
tools/nfsrahead/Makefile.am +1 -0
@@ -1,5 +1,6 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c +nfsrahead_LDFLAGS= -lmount udev_rulesdir = /etc/udev/rules.d udev_rules_DATA = 99-nfs_bdi.rules
tools/nfsrahead/main.c +130 -0
137 lines not on this page · raw message
Signature · 1 line
2.35.1
RFC PATCH v3 4/6 nfsrahead: add logging
In reply to Thiago BeckerBugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/Makefile.am | 1 + tools/nfsrahead/main.c | 40 +++++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-)
tools/nfsrahead/Makefile.am +1 -0
@@ -1,6 +1,7 @@ libexec_PROGRAMS = nfsrahead nfsrahead_SOURCES = main.c nfsrahead_LDFLAGS= -lmount +nfsrahead_LDADD = ../../support/nfs/libnfsconf.la udev_rulesdir = /etc/udev/rules.d udev_rules_DATA = 99-nfs_bdi.rules
tools/nfsrahead/main.c +34 -6
@@ -2,14 +2,19 @@#include <string.h> #include <stdlib.h> #include <errno.h> +#include <unistd.h> #include <libmount/libmount.h> #include <sys/sysmacros.h> +#include "xlog.h" +#ifndef MOUNTINFO_PATH #define MOUNTINFO_PATH "/proc/self/mountinfo"#endif +#define CONF_NAME "nfsrahead" +/* Device information from the system */ struct device_info {char *device_number;
@@ -108,26 +113,49 @@ static int get_device_info(const char *device_number, struct device_info *devicereturn ret; } +#define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL) +int main (int argc,char **argv,char **envp) {int ret =0 ;struct device_info device; -unsigned int readahead =128 ; - -if (argc !=2 ) { -return -EINVAL ; +unsigned int readahead =128 , verbose =0 , log_stderr =0 ; +char opt; + +while ((opt =getopt (argc, argv,"dF" )) !=-1 ) { +switch (opt) { +case 'd' : + verbose =1 ; +break ; +case 'F' : + log_stderr =1 ; +break ; + } } -if ((ret =get_device_info (argv[1 ], &device)) !=0 ) { +xlog_stderr (log_stderr); +xlog_syslog (~log_stderr); +xlog_config (L_DEFAULT | (L_NOTICE & verbose),1 ); +xlog_open (CONF_NAME ); + +// xlog_err causes the system to exit +if ((argc - optind) !=1 ) +xlog_err ("expected the device number of a BDI; is udev ok?" ); + +if ((ret =get_device_info (argv[optind], &device)) !=0 ) { +xlog (L_ERROR ,"unable to find device %s\n" , argv[optind]); goto out; }if (strncmp ("nfs" , device.fstype,3 ) !=0 ) { +xlog (L_NOTICE , +"not setting readahead for non supported fstype %s on device %s\n" , + device.fstype, argv[optind]); ret = -EINVAL ; goto out; } -info ("Setting %s readahead to 128\n" , device.mountpoint); +xlog (L_WARNING ,"setting %s readahead to %d\n" , device.mountpoint, readahead);printf ("%d\n" , readahead);
Signature · 1 line
2.35.1
RFC PATCH v3 5/6 hfsrahead: get the information from the config file.
In reply to Thiago BeckerBugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- tools/nfsrahead/main.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/nfsrahead/main.c +14 -0
@@ -8,12 +8,14 @@#include <sys/sysmacros.h> #include "xlog.h" +#include "conffile.h" #ifndef MOUNTINFO_PATH #define MOUNTINFO_PATH "/proc/self/mountinfo"#endif #define CONF_NAME "nfsrahead" +#define NFS_DEFAULT_READAHEAD 128/* Device information from the system */ struct device_info {
@@ -113,6 +115,14 @@ static int get_device_info(const char *device_number, struct device_info *devicereturn ret; } +static int conf_get_readahead (const char *kind) { +int readahead =0 ; + +if ((readahead =conf_get_num (CONF_NAME , kind,-1 )) ==-1 ) + readahead =conf_get_num (CONF_NAME ,"default" ,NFS_DEFAULT_READAHEAD ); + +return readahead; +}#define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL)int main (int argc,char **argv,char **envp)
@@ -133,6 +143,8 @@ int main(int argc, char **argv, char **envp) } } +conf_init_file (NFS_CONFFILE ); +xlog_stderr (log_stderr);xlog_syslog (~log_stderr);xlog_config (L_DEFAULT | (L_NOTICE & verbose),1 );
@@ -155,6 +167,8 @@ int main(int argc, char **argv, char **envp) goto out; } + readahead =conf_get_readahead (device.fstype); +xlog (L_WARNING ,"setting %s readahead to %d\n" , device.mountpoint, readahead);printf ("%d\n" , readahead);
Signature · 1 line
2.35.1
RFC PATCH v3 6/6 nfsrahead: User documentation
In reply to Thiago BeckerAdd the man page for nfsrahead, and add the new section to nfs.conf.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283
--- systemd/nfs.conf.man | 11 ++++++ tools/nfsrahead/Makefile.am | 2 + tools/nfsrahead/nfsrahead.man | 72 +++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+) create mode 100644 tools/nfsrahead/nfsrahead.man
systemd/nfs.conf.man +11 -0
@@ -294,6 +294,17 @@ Only .B debug= is recognized. +.TP +.B nfsrahead +Recognized values: +.BR nfs , +.BR nfsv4 , +.BR default . + +See +.BR nfsrahead (5) +for deatils. + .SH FILES .TP 10n .I /etc/nfs.conf
tools/nfsrahead/Makefile.am +2 -0
@@ -3,6 +3,8 @@ nfsrahead_SOURCES = main.c nfsrahead_LDFLAGS= -lmount nfsrahead_LDADD = ../../support/nfs/libnfsconf.la +man5_MANS = nfsrahead.man + udev_rulesdir = /etc/udev/rules.d udev_rules_DATA = 99-nfs_bdi.rules
tools/nfsrahead/nfsrahead.man new file +72 -0
72 lines not on this page · raw message
Signature · 1 line
2.35.1
Quoted 4 lines · Thiago Becker
Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.
Which recent changes? Something in NFS or something in the VFS/MM? Did you even think about asking a wider audience than the NFS mailing list? I only happened to notice this while I was looking for something else, otherwise I would never have seen it. The responses from other people to your patches were right; you're trying to do this all wrong.
Let's start out with a bug report instead of a solution. What changed and when?
Quoted 16 lines · Matthew Wilcox
Quoted 6 lines · Thiago Becker
Recent changes in the linux kernel caused NFS readahead to default to
128 from the previous default of 15 * rsize. This causes performance
penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.Which recent changes? Something in NFS or something in the VFS/MM? Did you even think about asking a wider audience than the NFS mailing list? I only happened to notice this while I was looking for something
else, otherwise I would never have seen it. The responses from other people to your patches were right; you're trying to do this all wrong.
Quoted 3 lines
Let's start out with a bug report instead of a solution. What changed
and when?
I believe Thiago is talking about the changes introduced by commit c128e575514c "NFS: Optimise the default readahead size" (i.e. we're talking about Linux 5.4).
...and yes, as the commit description notes, users who want to change the default can do so using the standard sysfs mechanism. AFAICS, all this is doing is providing a toolset to allow users to more easily set up and edit the udev scripts that will automate these settings.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Hello,
Quoted 9 lines · Matthew Wilcox
Which recent changes? Something in NFS or something in the VFS/MM? Did you even think about asking a wider audience than the NFS mailing list? I only happened to notice this while I was looking for something else, otherwise I would never have seen it. The responses from other people to your patches were right; you're trying to do this all wrong.
Let's start out with a bug report instead of a solution. What changed and when?
As Trond stated, c128e575514c ("NFS: Optimise the default readahead size") changed the way readahead is calculated for NFS mounts. This caused some read workloads to underperform, compared to the performance from previous revisions. To recall, the current policy is to adopt the system default readahead of 128kiB, and mounts with sec=krb5p take a performance hit of 50-75% when readahead is 128. I haven't performed an exhaustive search for other workloads that might also be affected, but I noticed the meaningful drop in performance in sec=sys mounts, notes at the end.
The previous policy was to calculate the readahead as a multiple of rsize, so we prescribed increasing the value to the complaining part, and this fixed the issue. We are now trying to find a solution that we can incorporate into the system.
thiago.
----- Tests
===== RAWHIDE (35% performance hit) =====
# uname -r 5.16.0-0.rc0.20211112git5833291ab6de.12.fc36.x86_64
# grep nfs /proc/self/mountinfo 601 60 0:55 / /mnt rw,relatime shared:332 - nfs4 192.168.122.225:/exports rw,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.83,local_lock=none,addr=192.168.122.225
# cat /sys/class/bdi/0\:55/read_ahead_kb 128
# for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5025 s, 260 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.4474 s, 261 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.0181 s, 238 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.2323 s, 236 MB/s# echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb
# for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2601 s, 381 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.1885 s, 384 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.5877 s, 371 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 10.9475 s, 392 MB/s===== UPSTREAM (30% performance hit) =====
# uname -r 5.17.0+
# grep nfs /proc/self/mountinfo 583 60 0:55 / /mnt rw,relatime shared:302 - nfs4 192.168.122.225:/exports rw,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.83,local_lock=none,addr=192.168.122.225
# cat /sys/class/bdi/0\:55/read_ahead_kb 128
# for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.056 s, 252 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.1258 s, 251 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5981 s, 259 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5487 s, 260 MB/s# echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb
# for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1
| grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.3855 s, 347 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2528 s, 382 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.9849 s, 358 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2953 s, 380 MB/s