Skip to contentSkip to conversation

Intruduce nfsrahead

PATCH In review Thiago Becker 2 participants 9 messages Apr 1, 2022 – Apr 19, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover Intruduce nfsrahead

Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.

Specifically, the read troughput on a sec=krb5p mount drops by 50-75% when comparing the default readahead with a readahead of 15360.

Previous discussions:
https://lore.kernel.org/linux-nfs/[email protected]/
I attempted to add a non-kernel option to mount.nfs, and it was rejected.

https://lore.kernel.org/linux-nfs/[email protected]/
Attempted to add a mount option to the kernel, rejected as well.

I had started a separate tool to set the readahead of BDIs, but the scope is specifically for NFS, so I would like to get the community feeling for having this in nfs-utils.

This patch series introduces nfs-readahead-udev, a utility to automatically set NFS readahead when NFS is mounted. The utility is triggered by udev when a new BDI is added, returns to udev the value of the readahead that should be used.

The tool currently supports setting read ahead per mountpoint, nfs major version, or by a global default value.

v2:
- explain the motivation

v3:
- adopt already available facilities
- nfsrahead is now configured in nfs.conf

v4:
- retry getting the device if it fails
- assorted fixes and improvements

Thiago Becker (7):
Create nfsrahead
nfsrahead: configure udev
nfsrahead: only set readahead for nfs devices.
nfsrahead: add logging
nfsrahead: get the information from the config file.
nfsrahead: User documentation
nfsrahead: retry getting the device if it fails.

 .gitignore                      |   2 +
 configure.ac                    |   1 +
 systemd/nfs.conf.man            |  11 ++
 tools/Makefile.am               |   2 +-
 tools/nfsrahead/99-nfs.rules.in |   1 +
 tools/nfsrahead/Makefile.am     |  16 +++
 tools/nfsrahead/main.c          | 183 ++++++++++++++++++++++++++++++++
 tools/nfsrahead/nfsrahead.man   |  72 +++++++++++++
 8 files changed, 287 insertions(+), 1 deletion(-)
 create mode 100644 tools/nfsrahead/99-nfs.rules.in
 create mode 100644 tools/nfsrahead/Makefile.am
 create mode 100644 tools/nfsrahead/main.c
 create mode 100644 tools/nfsrahead/nfsrahead.man
Signature · 1 line
2.35.1
#1 lore raw

PATCH v4 1/7 Create nfsrahead

Kernel commit c128e575514c ("NFS: Optimise the default readahead size") changed the calculation for NFS readahead from a multiple of rsize to the system default, 128 kiB. This setting has been causing read heavy workloads to underperform by at least 30%, as show below.

$ cat /sys/class/bdi/0\:55/read_ahead_kb
128
$ for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 \
      | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.056 s, 252 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.1258 s, 251 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5981 s, 259 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5487 s, 260 MB/s
$ echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb
$ for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 \
      | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.3855 s, 347 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2528 s, 382 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.9849 s, 358 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2953 s, 380 MB/s

This patch and the following create a tool for automatically setting NFS readahead during the mount, nfsrahead. The tool is invoked from udev when the NFS backing device is created in kernel, and sets the readahead using the sysfs interface.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 .gitignore                  | 1 +
 configure.ac                | 1 +
 tools/Makefile.am           | 2 +-
 tools/nfsrahead/Makefile.am | 3 +++
 tools/nfsrahead/main.c      | 7 +++++++
 5 files changed, 13 insertions(+), 1 deletion(-)
 create mode 100644 tools/nfsrahead/Makefile.am
 create mode 100644 tools/nfsrahead/main.c
.gitignore +1 -0
index c89d1cd2..38ab1d39 100644
@@ -61,6 +61,7 @@ utils/statd/statd
 tools/locktest/testlk
 tools/getiversion/getiversion
 tools/nfsconf/nfsconf
+tools/nfsrahead/nfsrahead
 support/export/mount.h
 support/export/mount_clnt.c
 support/export/mount_xdr.c
configure.ac +1 -0
index e0f5a930..3e1c183b 100644
@@ -737,6 +737,7 @@ AC_CONFIG_FILES([
 	tools/rpcgen/Makefile
 	tools/mountstats/Makefile
 	tools/nfs-iostat/Makefile
+	tools/nfsrahead/Makefile
 	tools/rpcctl/Makefile
 	tools/nfsdclnts/Makefile
 	tools/nfsconf/Makefile
tools/Makefile.am +1 -1
index c3feabbe..40c17c37 100644
@@ -12,6 +12,6 @@ if CONFIG_NFSDCLD
 OPTDIRS += nfsdclddb
 endif
 
-SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts $(OPTDIRS)
+SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat rpcctl nfsdclnts nfsrahead $(OPTDIRS)
 
 MAINTAINERCLEANFILES = Makefile.in
tools/nfsrahead/Makefile.am new file +3 -0
new file mode 100644
index 00000000..edff7921
@@ -0,0 +1,3 @@
+libexec_PROGRAMS = nfsrahead
+nfsrahead_SOURCES = main.c
+
tools/nfsrahead/main.c new file +7 -0
new file mode 100644
index 00000000..0359aced
@@ -0,0 +1,7 @@
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+	unsigned int readahead = 128;
+	printf("%d\n", readahead);
+}
Signature · 1 line
2.35.1
#2 lore raw

PATCH v4 2/7 nfsrahead: configure udev

In reply to Thiago Becker

Set the udev rule to call the readahead utility.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 .gitignore                      | 1 +
 tools/nfsrahead/99-nfs.rules.in | 1 +
 tools/nfsrahead/Makefile.am     | 8 ++++++++
 3 files changed, 10 insertions(+)
 create mode 100644 tools/nfsrahead/99-nfs.rules.in
.gitignore +1 -0
index 38ab1d39..df791a83 100644
@@ -62,6 +62,7 @@ tools/locktest/testlk
 tools/getiversion/getiversion
 tools/nfsconf/nfsconf
 tools/nfsrahead/nfsrahead
+tools/nfsrahead/99-nfs_bdi.rules
 support/export/mount.h
 support/export/mount_clnt.c
 support/export/mount_xdr.c
tools/nfsrahead/99-nfs.rules.in new file +1 -0
new file mode 100644
index 00000000..7d55b407
@@ -0,0 +1 @@
+SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead", ATTR{read_ahead_kb}="%c"
tools/nfsrahead/Makefile.am +8 -0
index edff7921..58a2ea29 100644
@@ -1,3 +1,11 @@
 libexec_PROGRAMS = nfsrahead
 nfsrahead_SOURCES = main.c
 
+udev_rulesdir = /usr/lib/udev/rules.d/
+udev_rules_DATA = 99-nfs.rules
+
+99-nfs.rules: 99-nfs.rules.in $(builddefs)
+	$(SED) "s|_libexecdir_|@libexecdir@|g" 99-nfs.rules.in > $@
+
+clean-local:
+	$(RM) 99-nfs.rules
Signature · 1 line
2.35.1
#3 lore raw

PATCH v4 3/7 nfsrahead: only set readahead for nfs devices.

In reply to Thiago Becker

Limit setting the readahead for nfs devices.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 tools/nfsrahead/99-nfs.rules.in |   2 +-
 tools/nfsrahead/Makefile.am     |   1 +
 tools/nfsrahead/main.c          | 128 ++++++++++++++++++++++++++++++++
 3 files changed, 130 insertions(+), 1 deletion(-)
tools/nfsrahead/99-nfs.rules.in +1 -1
index 7d55b407..648813c5 100644
@@ -1 +1 @@
-SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead", ATTR{read_ahead_kb}="%c"
+SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="_libexecdir_/nfsrahead %k", ATTR{read_ahead_kb}="%c"
tools/nfsrahead/Makefile.am +1 -0
index 58a2ea29..0daddc4b 100644
@@ -1,5 +1,6 @@
 libexec_PROGRAMS = nfsrahead
 nfsrahead_SOURCES = main.c
+nfsrahead_LDFLAGS= -lmount
 
 udev_rulesdir = /usr/lib/udev/rules.d/
 udev_rules_DATA = 99-nfs.rules
tools/nfsrahead/main.c +128 -0

135 lines not on this page · raw message

Signature · 1 line
2.35.1
#4 lore raw

PATCH v4 4/7 nfsrahead: add logging

In reply to Thiago Becker

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 tools/nfsrahead/Makefile.am |  1 +
 tools/nfsrahead/main.c      | 40 ++++++++++++++++++++++++++++++++-----
 2 files changed, 36 insertions(+), 5 deletions(-)
tools/nfsrahead/Makefile.am +1 -0
index 0daddc4b..60a1102a 100644
@@ -1,6 +1,7 @@
 libexec_PROGRAMS = nfsrahead
 nfsrahead_SOURCES = main.c
 nfsrahead_LDFLAGS= -lmount
+nfsrahead_LDADD = ../../support/nfs/libnfsconf.la
 
 udev_rulesdir = /usr/lib/udev/rules.d/
 udev_rules_DATA = 99-nfs.rules
tools/nfsrahead/main.c +35 -5
index 8fe6d648..b630b92f 100644
@@ -2,14 +2,19 @@
 #include <string.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <unistd.h>
 
 #include <libmount/libmount.h>
 #include <sys/sysmacros.h>
 
+#include "xlog.h"
+
 #ifndef MOUNTINFO_PATH
 #define MOUNTINFO_PATH "/proc/self/mountinfo"
 #endif
 
+#define CONF_NAME "nfsrahead"
+
 /* Device information from the system */
 struct device_info {
 	char *device_number;
@@ -108,25 +113,50 @@ static int get_device_info(const char *device_number, struct device_info *device
 	return ret;
 }
 
+#define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL)
+
 int main(int argc, char **argv)
 {
 	int ret = 0;
 	struct device_info device;
-	unsigned int readahead = 128;
-
-	if (argc != 2) {
-		return -EINVAL;
+	unsigned int readahead = 128, verbose = 0, log_stderr = 0;
+	char opt;
+
+	while((opt = getopt(argc, argv, "dF")) != -1) {
+		switch (opt) {
+		case 'd':
+			verbose = 1;
+			break;
+		case 'F':
+			log_stderr = 1;
+			break;
+		}
 	}
 
-	if ((ret = get_device_info(argv[1], &device)) != 0) {
+	xlog_stderr(log_stderr);
+	xlog_syslog(~log_stderr);
+	xlog_config(L_DEFAULT | (L_NOTICE & verbose), 1);
+	xlog_open(CONF_NAME);
+
+	// xlog_err causes the system to exit
+	if ((argc - optind) != 1)
+		xlog_err("expected the device number of a BDI; is udev ok?");
+
+	if ((ret = get_device_info(argv[optind], &device)) != 0) {
+		xlog(L_ERROR, "unable to find device %s\n", argv[optind]);
 		goto out;
 	}
 
 	if (strncmp("nfs", device.fstype, 3) != 0) {
+		xlog(L_NOTICE,
+			"not setting readahead for non supported fstype %s on device %s\n",
+			device.fstype, argv[optind]);
 		ret = -EINVAL;
 		goto out;
 	}
 
+	xlog(L_WARNING, "setting %s readahead to %d\n", device.mountpoint, readahead);
+
 	printf("%d\n", readahead);
 
 out:
Signature · 1 line
2.35.1
#5 lore raw

PATCH v4 5/7 nfsrahead: get the information from the config file.

In reply to Thiago Becker

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 tools/nfsrahead/main.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
tools/nfsrahead/main.c +14 -0
index b630b92f..1a312ac9 100644
@@ -8,12 +8,14 @@
 #include <sys/sysmacros.h>
 
 #include "xlog.h"
+#include "conffile.h"
 
 #ifndef MOUNTINFO_PATH
 #define MOUNTINFO_PATH "/proc/self/mountinfo"
 #endif
 
 #define CONF_NAME "nfsrahead"
+#define NFS_DEFAULT_READAHEAD 128
 
 /* Device information from the system */
 struct device_info {
@@ -113,6 +115,14 @@ static int get_device_info(const char *device_number, struct device_info *device
 	return ret;
 }
 
+static int conf_get_readahead(const char *kind) {
+	int readahead = 0;
+
+	if((readahead = conf_get_num(CONF_NAME, kind, -1)) == -1)
+		readahead = conf_get_num(CONF_NAME, "default", NFS_DEFAULT_READAHEAD);
+	
+	return readahead;
+}
 #define L_DEFAULT (L_WARNING | L_ERROR | L_FATAL)
 
 int main(int argc, char **argv)
@@ -133,6 +143,8 @@ int main(int argc, char **argv)
 		}
 	}
 
+	conf_init_file(NFS_CONFFILE);
+
 	xlog_stderr(log_stderr);
 	xlog_syslog(~log_stderr);
 	xlog_config(L_DEFAULT | (L_NOTICE & verbose), 1);
@@ -155,6 +167,8 @@ int main(int argc, char **argv)
 		goto out;
 	}
 
+	readahead = conf_get_readahead(device.fstype);
+
 	xlog(L_WARNING, "setting %s readahead to %d\n", device.mountpoint, readahead);
 
 	printf("%d\n", readahead);
Signature · 1 line
2.35.1
#6 lore raw

PATCH v4 6/7 nfsrahead: User documentation

In reply to Thiago Becker

Add the man page for nfsrahead, and add the new section to nfs.conf.

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1946283

---
 systemd/nfs.conf.man          | 11 ++++++
 tools/nfsrahead/Makefile.am   |  3 ++
 tools/nfsrahead/nfsrahead.man | 72 +++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+)
 create mode 100644 tools/nfsrahead/nfsrahead.man
systemd/nfs.conf.man +11 -0
index be487a11..e74083e9 100644
@@ -294,6 +294,17 @@ Only
 .B debug=
 is recognized.
 
+.TP
+.B nfsrahead
+Recognized values:
+.BR nfs ,
+.BR nfsv4 ,
+.BR default .
+
+See
+.BR nfsrahead (5)
+for deatils.
+
 .SH FILES
 .TP 10n
 .I /etc/nfs.conf
tools/nfsrahead/Makefile.am +3 -0
index 60a1102a..845ea0d5 100644
@@ -3,6 +3,9 @@ nfsrahead_SOURCES = main.c
 nfsrahead_LDFLAGS= -lmount
 nfsrahead_LDADD = ../../support/nfs/libnfsconf.la
 
+man5_MANS = nfsrahead.man
+EXTRA_DIST = $(man5_MANS)
+
 udev_rulesdir = /usr/lib/udev/rules.d/
 udev_rules_DATA = 99-nfs.rules
 
tools/nfsrahead/nfsrahead.man new file +72 -0

72 lines not on this page · raw message

Signature · 1 line
2.35.1
#7 lore raw

PATCH v4 7/7 nfsrahead: retry getting the device if it fails.

In reply to Thiago Becker

Sometimes the mountinfo entry is not available when nfsrahead is called, leading to failure to set the readahead. Retry getting the device before failing.

---
 tools/nfsrahead/main.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
tools/nfsrahead/main.c +6 -2
index 1a312ac9..b3af3aa8 100644
@@ -127,7 +127,7 @@ static int conf_get_readahead(const char *kind) {
 
 int main(int argc, char **argv)
 {
-	int ret = 0;
+	int ret = 0, retry;
 	struct device_info device;
 	unsigned int readahead = 128, verbose = 0, log_stderr = 0;
 	char opt;
@@ -154,7 +154,11 @@ int main(int argc, char **argv)
 	if ((argc - optind) != 1)
 		xlog_err("expected the device number of a BDI; is udev ok?");
 
-	if ((ret = get_device_info(argv[optind], &device)) != 0) {
+	for (retry = 0; retry <= 10; retry++ )
+		if ((ret = get_device_info(argv[optind], &device)) == 0)
+			break;
+
+	if (ret != 0) {
 		xlog(L_ERROR, "unable to find device %s\n", argv[optind]);
 		goto out;
 	}
Signature · 1 line
2.35.1
#8 lore raw
In reply to Thiago Becker
Quoted 57 lines · Thiago Becker

Recent changes in the linux kernel caused NFS readahead to default to 128 from the previous default of 15 * rsize. This causes performance penalties to some read-heavy workloads, which can be fixed by tuning the readahead for that given mount.

Specifically, the read troughput on a sec=krb5p mount drops by 50-75% when comparing the default readahead with a readahead of 15360.

Previous discussions:
https://lore.kernel.org/linux-nfs/[email protected]/
I attempted to add a non-kernel option to mount.nfs, and it was rejected.

https://lore.kernel.org/linux-nfs/[email protected]/
Attempted to add a mount option to the kernel, rejected as well.

I had started a separate tool to set the readahead of BDIs, but the scope is specifically for NFS, so I would like to get the community feeling for having this in nfs-utils.

This patch series introduces nfs-readahead-udev, a utility to automatically set NFS readahead when NFS is mounted. The utility is triggered by udev when a new BDI is added, returns to udev the value of the readahead that should be used.

The tool currently supports setting read ahead per mountpoint, nfs major version, or by a global default value.

v2:
- explain the motivation

v3:
- adopt already available facilities
- nfsrahead is now configured in nfs.conf

v4:
- retry getting the device if it fails
- assorted fixes and improvements

Thiago Becker (7):
Create nfsrahead
nfsrahead: configure udev
nfsrahead: only set readahead for nfs devices.
nfsrahead: add logging
nfsrahead: get the information from the config file.
nfsrahead: User documentation
nfsrahead: retry getting the device if it fails.

  .gitignore                      |   2 +
  configure.ac                    |   1 +
  systemd/nfs.conf.man            |  11 ++
  tools/Makefile.am               |   2 +-
  tools/nfsrahead/99-nfs.rules.in |   1 +
  tools/nfsrahead/Makefile.am     |  16 +++
  tools/nfsrahead/main.c          | 183 ++++++++++++++++++++++++++++++++
  tools/nfsrahead/nfsrahead.man   |  72 +++++++++++++
  8 files changed, 287 insertions(+), 1 deletion(-)
Quoted 5 lines

create mode 100644 tools/nfsrahead/99-nfs.rules.in create mode 100644 tools/nfsrahead/Makefile.am create mode 100644 tools/nfsrahead/main.c create mode 100644 tools/nfsrahead/nfsrahead.man

Committed... (tag nfs-utils-2-6-2-rc4)

steved.