Skip to contentSkip to conversation

nfs-utils: Improving NFS re-export wrt. crossmnt

PATCH In review Richard Weinberger 2 participants 10 messages Apr 18, 2023 – Apr 19, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover nfs-utils: Improving NFS re-export wrt. crossmnt

After a longer hiatus I'm sending the next iteration of my re-export improvement patch series. While the kernel side is upstream since v6.2, the nfs-utils parts are still missing.
This patch series aims to solve this.

The core idea is adding new export option, reeport=
Using reexport= it is possible to mark an export entry in the exports file explicitly as NFS re-export and select a strategy on how unique identifiers should be provided. This makes the crossmnt feature work in the re-export case.
Currently two strategies are supported, "auto-fsidnum" and "predefined-fsidnum".

In my earlier series a sqlite database was mandatory to keep track of generated fsids.
This series follows a different approach, instead of directly using sqlite in all nfs-utils components (linking libsqlite), a new deamon manages the database, fsidd.
fsidd offers a simple (but stupid?) text based interface over a unix domain socket which can be queried by mountd, exportfs, etc. for fsidnums. The main idea behind fsidd is allowing users to implement their own fsidd which keeps global state across load balancers.
I'm still not happy with fsidd, there is room for improvement but first I'd like to know whether you like or hate this approach.

A typical export entry on a re-exporting server looks like:
/nfs *(rw,no_root_squash,no_subtree_check,crossmnt,reexport=auto-fsidnum)
reexport=auto-fsidnum will automatically assign an fsid= to /nfs and all uncovered subvolumes.

Changes since v2, https://lore.kernel.org/linux-nfs/[email protected]/
- Split patch series
- Add improved fsidd system unit file
- Rebased to nfs-utils master as of today
- Dropped init code from exportd

Changes since v1, https://lore.kernel.org/linux-nfs/[email protected]/
- Factor out Sqlite and put it into a daemon
- Add fsidd
- Basically re-implemented the patch series
- Lot's of fixes (e.g. nfs v4 root export)

Richard Weinberger (8):
Add reexport helper library
Implement reexport= export option
export: Wireup reexport mechanism
export: Uncover NFS subvolume after reboot
exports.man: Document reexport= option
reexport: Add sqlite backend
export: Add fsidd
Add fsid systemd service file

 configure.ac                        |   1 +
 support/Makefile.am                 |   2 +-
 support/export/Makefile.am          |   2 +
 support/export/cache.c              |  74 ++++++-
 support/export/export.c             |  20 ++
 support/include/nfslib.h            |   1 +
 support/nfs/Makefile.am             |   1 +
 support/nfs/exports.c               |  62 ++++++
 support/reexport/Makefile.am        |  18 ++
 support/reexport/backend_sqlite.c   | 267 +++++++++++++++++++++++
 support/reexport/fsidd.c            | 198 +++++++++++++++++
 support/reexport/reexport.c         | 326 ++++++++++++++++++++++++++++
 support/reexport/reexport.h         |  18 ++
 support/reexport/reexport_backend.h |  47 ++++
 systemd/Makefile.am                 |   5 +-
 systemd/fsidd.service               |  10 +
 utils/exportd/Makefile.am           |   4 +-
 utils/exportfs/Makefile.am          |   3 +
 utils/exportfs/exportfs.c           |  11 +
 utils/exportfs/exports.man          |  31 +++
 utils/mount/Makefile.am             |   3 +-
 utils/mountd/Makefile.am            |   2 +
 22 files changed, 1096 insertions(+), 10 deletions(-)
 create mode 100644 support/reexport/Makefile.am
 create mode 100644 support/reexport/backend_sqlite.c
 create mode 100644 support/reexport/fsidd.c
 create mode 100644 support/reexport/reexport.c
 create mode 100644 support/reexport/reexport.h
 create mode 100644 support/reexport/reexport_backend.h
 create mode 100644 systemd/fsidd.service
Signature · 1 line
2.31.1
#1 lore raw

PATCH v3 1/8 Add reexport helper library

Add some helper functions which will be used by the reexport mechanism to create and find fsidnums for re-exported NFS shares.

---
 configure.ac                 |   1 +
 support/Makefile.am          |   2 +-
 support/export/Makefile.am   |   2 +
 support/include/nfslib.h     |   1 +
 support/reexport/Makefile.am |   6 +
 support/reexport/reexport.c  | 326 +++++++++++++++++++++++++++++++++++
 support/reexport/reexport.h  |  18 ++
 7 files changed, 355 insertions(+), 1 deletion(-)
 create mode 100644 support/reexport/Makefile.am
 create mode 100644 support/reexport/reexport.c
 create mode 100644 support/reexport/reexport.h
configure.ac +1 -0
index 7672a760..9f43267c 100644
@@ -717,6 +717,7 @@ AC_CONFIG_FILES([
 	support/nsm/Makefile
 	support/nfsidmap/Makefile
 	support/nfsidmap/libnfsidmap.pc
+	support/reexport/Makefile
 	tools/Makefile
 	tools/locktest/Makefile
 	tools/nlmtest/Makefile
support/Makefile.am +1 -1
index c962d4d4..07cfd87e 100644
@@ -10,7 +10,7 @@ if CONFIG_JUNCTION
 OPTDIRS += junction
 endif
 
-SUBDIRS = export include misc nfs nsm $(OPTDIRS)
+SUBDIRS = export include misc nfs nsm reexport $(OPTDIRS)
 
 MAINTAINERCLEANFILES = Makefile.in
 
support/export/Makefile.am +2 -0
index eec737f6..7338e1c7 100644
@@ -14,6 +14,8 @@ libexport_a_SOURCES = client.c export.c hostname.c \
 		      xtab.c mount_clnt.c mount_xdr.c \
 		      cache.c auth.c v4root.c fsloc.c \
 		      v4clients.c
+libexport_a_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) -I$(top_srcdir)/support/reexport
+
 BUILT_SOURCES 	= $(GENFILES)
 
 noinst_HEADERS = mount.h
support/include/nfslib.h +1 -0
index 61c19933..bdbde78d 100644
@@ -98,6 +98,7 @@ struct exportent {
 	struct xprtsec_entry e_xprtsec[XPRTSECMODE_COUNT + 1];
 	unsigned int	e_ttl;
 	char *		e_realpath;
+	int		e_reexport;
 };
 
 struct rmtabent {
support/reexport/Makefile.am new file +6 -0
new file mode 100644
index 00000000..9d544a8f
@@ -0,0 +1,6 @@
+## Process this file with automake to produce Makefile.in
+
+noinst_LIBRARIES = libreexport.a
+libreexport_a_SOURCES = reexport.c
+
+MAINTAINERCLEANFILES = Makefile.in
support/reexport/reexport.c new file +326 -0

326 lines not on this page · raw message

support/reexport/reexport.h new file +18 -0
new file mode 100644
index 00000000..3bed03a9
@@ -0,0 +1,18 @@
+#ifndef REEXPORT_H
+#define REEXPORT_H
+
+enum {
+	REEXP_NONE = 0,
+	REEXP_AUTO_FSIDNUM,
+	REEXP_PREDEFINED_FSIDNUM,
+};
+
+int reexpdb_init(void);
+void reexpdb_destroy(void);
+int reexpdb_fsidnum_by_path(char *path, uint32_t *fsidnum, int may_create);
+int reexpdb_apply_reexport_settings(struct exportent *ep, char *flname, int flline);
+void reexpdb_uncover_subvolume(uint32_t fsidnum);
+
+#define FSID_SOCKET_NAME "fsid.sock"
+
+#endif /* REEXPORT_H */
Signature · 1 line
2.31.1
#2 lore raw

PATCH v3 2/8 Implement reexport= export option

In reply to Richard Weinberger

When re-exporting a NFS volume it is mandatory to specify either a UUID or numerical fsid= option because nfsd is unable to derive an identifier on its own.

For NFS cross mounts this becomes a problem because nfsd also needs an identifier for every crossed mount.
A common workaround is stating every single subvolume in the exports list too.
But this defeats the purpose of the crossmnt option and is tedious.

This is where the reexport= tries to help.
It offers various strategies to automatically derive a identifier for NFS volumes and sub volumes.

Currently two strategies are implemented:

1. auto-fsidnum
In this mode mountd/exportd will create a new numerical fsid for a NFS volume and subvolume. The numbers are stored in a database, via fsidd, such that the server will always use the same fsid. The entry in the exports file allowed to skip the fsid= option but stating a UUID is allowed, if needed.

This mode has the obvious downside that load balancing is by default not possible since multiple re-exporting NFS servers would generate different ids.
It is possible if all load balancers use the same database. This can be achieved by using nfs-utils' fsidd and placing it's sqlit database on a network share which supports file locks or by implementing your own fsidd which is able to provide consistent fsids across multiple re-exporting nfs servers.

2. predefined-fsidnum
This mode works just like auto-fsidnum but does not generate ids for you. It helps in the load balancing case. A system administrator has to manually maintain the database and install it on all re-exporting NFS servers. If you have a massive amount of subvolumes this mode will help because you don't have to bloat the exports list.

---
 support/export/export.c    | 20 ++++++++++++
 support/nfs/Makefile.am    |  1 +
 support/nfs/exports.c      | 62 ++++++++++++++++++++++++++++++++++++++
 systemd/Makefile.am        |  2 ++
 utils/exportd/Makefile.am  |  4 ++-
 utils/exportfs/Makefile.am |  3 ++
 utils/exportfs/exportfs.c  | 11 +++++++
 utils/mount/Makefile.am    |  3 +-
 utils/mountd/Makefile.am   |  2 ++
 9 files changed, 106 insertions(+), 2 deletions(-)
support/export/export.c +20 -0
index 03390dfc..3e48c42d 100644
@@ -25,6 +25,7 @@
 #include "exportfs.h"
 #include "nfsd_path.h"
 #include "xlog.h"
+#include "reexport.h"
 
 exp_hash_table exportlist[MCL_MAXTYPES] = {{NULL, {{NULL,NULL}, }}, }; 
 static int export_hash(char *);
@@ -115,6 +116,7 @@ export_read(char *fname, int ignore_hosts)
 	nfs_export		*exp;
 
 	int volumes = 0;
+	int reexport_found = 0;
 
 	setexportent(fname, "r");
 	while ((eep = getexportent(0,1)) != NULL) {
@@ -126,7 +128,25 @@ export_read(char *fname, int ignore_hosts)
 		}
 		else
 			warn_duplicated_exports(exp, eep);
+
+		if (eep->e_reexport)
+			reexport_found = 1;
 	}
+
+	if (reexport_found) {
+		for (int i = 0; i < MCL_MAXTYPES; i++) {
+			for (exp = exportlist[i].p_head; exp; exp = exp->m_next) {
+				if (exp->m_export.e_reexport)
+					continue;
+
+				if (exp->m_export.e_flags & NFSEXP_FSID) {
+					xlog(L_ERROR, "When a reexport= option is present no manully assigned numerical fsid= options are allowed");
+					return -1;
+				}
+			}
+		}
+	}
+
 	endexportent();
 
 	return volumes;
support/nfs/Makefile.am +1 -0
index 67e3a8e1..2e1577cc 100644
@@ -9,6 +9,7 @@ libnfs_la_SOURCES = exports.c rmtab.c xio.c rpcmisc.c rpcdispatch.c \
 		   svc_socket.c cacheio.c closeall.c nfs_mntent.c \
 		   svc_create.c atomicio.c strlcat.c strlcpy.c
 libnfs_la_LIBADD = libnfsconf.la
+libnfs_la_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) -I$(top_srcdir)/support/reexport
 
 libnfsconf_la_SOURCES = conffile.c xlog.c
 
support/nfs/exports.c +62 -0

98 lines not on this page · raw message

systemd/Makefile.am +2 -0
index 577c6a22..2e250dca 100644
@@ -70,8 +70,10 @@ rpc_pipefs_generator_SOURCES = $(COMMON_SRCS) rpc-pipefs-generator.c
 nfs_server_generator_LDADD = ../support/export/libexport.a \
 			     ../support/nfs/libnfs.la \
 			     ../support/misc/libmisc.a \
+			     ../support/reexport/libreexport.a \
 			     $(LIBPTHREAD)
 
+
 rpc_pipefs_generator_LDADD = ../support/nfs/libnfs.la
 
 if INSTALL_SYSTEMD
utils/exportd/Makefile.am +3 -1
index c95bdee7..83024958 100644
@@ -16,7 +16,9 @@ exportd_SOURCES = exportd.c
 exportd_LDADD = ../../support/export/libexport.a \
 			../../support/nfs/libnfs.la \
 			../../support/misc/libmisc.a \
-			$(OPTLIBS) $(LIBBLKID) $(LIBPTHREAD) -luuid
+			../support/reexport/libreexport.a \
+			$(OPTLIBS) $(LIBBLKID) $(LIBPTHREAD) \
+			-luuid
 
 exportd_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) \
 		-I$(top_srcdir)/support/export
utils/exportfs/Makefile.am +3 -0
index 96524c72..7f8ce9fa 100644
@@ -10,6 +10,9 @@ exportfs_SOURCES = exportfs.c
 exportfs_LDADD = ../../support/export/libexport.a \
 	       	 ../../support/nfs/libnfs.la \
 		 ../../support/misc/libmisc.a \
+		 ../../support/reexport/libreexport.a \
 		 $(LIBWRAP) $(LIBNSL) $(LIBPTHREAD)
 
+exportfs_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) -I$(top_srcdir)/support/reexport
+
 MAINTAINERCLEANFILES = Makefile.in
utils/exportfs/exportfs.c +11 -0
index 37b9e4b3..b03a047b 100644
@@ -38,6 +38,7 @@
 #include "exportfs.h"
 #include "xlog.h"
 #include "conffile.h"
+#include "reexport.h"
 
 static void	export_all(int verbose);
 static void	exportfs(char *arg, char *options, int verbose);
@@ -719,6 +720,16 @@ dump(int verbose, int export_format)
 				c = dumpopt(c, "fsid=%d", ep->e_fsid);
 			if (ep->e_uuid)
 				c = dumpopt(c, "fsid=%s", ep->e_uuid);
+			if (ep->e_reexport) {
+				switch (ep->e_reexport) {
+					case REEXP_AUTO_FSIDNUM:
+						c = dumpopt(c, "reexport=%s", "auto-fsidnum");
+						break;
+					case REEXP_PREDEFINED_FSIDNUM:
+						c = dumpopt(c, "reexport=%s", "predefined-fsidnum");
+						break;
+				}
+			}
 			if (ep->e_mountpoint)
 				c = dumpopt(c, "mountpoint%s%s",
 					    ep->e_mountpoint[0]?"=":"",
utils/mount/Makefile.am +2 -1
index 3101f7ab..5ff1148c 100644
@@ -29,8 +29,9 @@ endif
 
 mount_nfs_LDADD = ../../support/nfs/libnfs.la \
 		  ../../support/export/libexport.a \
+		  ../../support/reexport/libreexport.a \
 		  ../../support/misc/libmisc.a \
-		  $(LIBTIRPC)
+		   $(LIBTIRPC) $(LIBPTHREAD)
 
 mount_nfs_SOURCES = $(mount_common)
 
utils/mountd/Makefile.am +2 -0
index 13b25c90..197ef29b 100644
@@ -17,9 +17,11 @@ mountd_SOURCES = mountd.c mount_dispatch.c rmtab.c \
 mountd_LDADD = ../../support/export/libexport.a \
 	       ../../support/nfs/libnfs.la \
 	       ../../support/misc/libmisc.a \
+	       ../../support/reexport/libreexport.a \
 	       $(OPTLIBS) \
 	       $(LIBBSD) $(LIBWRAP) $(LIBNSL) $(LIBBLKID) -luuid $(LIBTIRPC) \
 	       $(LIBPTHREAD)
+
 mountd_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) \
 		  -I$(top_builddir)/support/include \
 		  -I$(top_srcdir)/support/export
Signature · 1 line
2.31.1
#3 lore raw

PATCH v3 3/8 export: Wireup reexport mechanism

In reply to Richard Weinberger

Detect the case when a NFS share is re-exported and assign an fsidnum to it.
The fsidnum is read (or created) from the reexport database.

---
 support/export/cache.c | 68 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 62 insertions(+), 6 deletions(-)
support/export/cache.c +62 -6
index 0a37703b..42a694d0 100644
@@ -33,6 +33,7 @@
 #include "export.h"
 #include "pseudoflavors.h"
 #include "xcommon.h"
+#include "reexport.h"
 
 #ifdef HAVE_JUNCTION_SUPPORT
 #include "fsloc.h"
@@ -235,6 +236,16 @@ static void auth_unix_gid(int f)
 		xlog(L_ERROR, "auth_unix_gid: error writing reply");
 }
 
+static int match_crossmnt_fsidnum(uint32_t parsed_fsidnum, char *path)
+{
+	uint32_t fsidnum;
+
+	if (reexpdb_fsidnum_by_path(path, &fsidnum, 0) == 0)
+		return 0;
+
+	return fsidnum == parsed_fsidnum;
+}
+
 #ifdef USE_BLKID
 static const char *get_uuid_blkdev(char *path)
 {
@@ -688,8 +699,13 @@ static int match_fsid(struct parsed_fsid *parsed, nfs_export *exp, char *path)
 		goto match;
 	case FSID_NUM:
 		if (((exp->m_export.e_flags & NFSEXP_FSID) == 0 ||
-		     exp->m_export.e_fsid != parsed->fsidnum))
+		     exp->m_export.e_fsid != parsed->fsidnum)) {
+			if ((exp->m_export.e_flags & NFSEXP_CROSSMOUNT) && exp->m_export.e_reexport != REEXP_NONE &&
+			    match_crossmnt_fsidnum(parsed->fsidnum, path))
+				goto match;
+
 			goto nomatch;
+		}
 		goto match;
 	case FSID_UUID4_INUM:
 	case FSID_UUID16_INUM:
@@ -937,7 +953,7 @@ static void write_fsloc(char **bp, int *blen, struct exportent *ep)
 }
 #endif
 
-static void write_secinfo(char **bp, int *blen, struct exportent *ep, int flag_mask)
+static void write_secinfo(char **bp, int *blen, struct exportent *ep, int flag_mask, int extra_flag)
 {
 	struct sec_entry *p;
 
@@ -952,7 +968,7 @@ static void write_secinfo(char **bp, int *blen, struct exportent *ep, int flag_m
 	qword_addint(bp, blen, p - ep->e_secinfo);
 	for (p = ep->e_secinfo; p->flav; p++) {
 		qword_addint(bp, blen, p->flav->fnum);
-		qword_addint(bp, blen, p->flags & flag_mask);
+		qword_addint(bp, blen, (p->flags | extra_flag) & flag_mask);
 	}
 }
 
@@ -970,6 +986,15 @@ static void write_xprtsec(char **bp, int *blen, struct exportent *ep)
 		qword_addint(bp, blen, p->info->number);
 }
 
+static int can_reexport_via_fsidnum(struct exportent *exp, struct statfs *st)
+{
+	if (st->f_type != 0x6969 /* NFS_SUPER_MAGIC */)
+		return 0;
+
+	return exp->e_reexport == REEXP_PREDEFINED_FSIDNUM ||
+	       exp->e_reexport == REEXP_AUTO_FSIDNUM;
+}
+
 static int dump_to_cache(int f, char *buf, int blen, char *domain,
 			 char *path, struct exportent *exp, int ttl)
 {
@@ -986,17 +1011,48 @@ static int dump_to_cache(int f, char *buf, int blen, char *domain,
 	if (exp) {
 		int different_fs = strcmp(path, exp->e_path) != 0;
 		int flag_mask = different_fs ? ~NFSEXP_FSID : ~0;
+		int rc, do_fsidnum = 0;
+		uint32_t fsidnum = exp->e_fsid;
+
+		if (different_fs) {
+			struct statfs st;
+
+			rc = nfsd_path_statfs(path, &st);
+			if (rc) {
+				xlog(L_WARNING, "unable to statfs %s", path);
+				errno = EINVAL;
+				return -1;
+			}
+
+			if (can_reexport_via_fsidnum(exp, &st)) {
+				do_fsidnum = 1;
+				flag_mask = ~0;
+			}
+		}
 
 		qword_adduint(&bp, &blen, now + exp->e_ttl);
-		qword_addint(&bp, &blen, exp->e_flags & flag_mask);
+
+		if (do_fsidnum) {
+			uint32_t search_fsidnum = 0;
+			if (exp->e_reexport != REEXP_NONE && reexpdb_fsidnum_by_path(path, &search_fsidnum,
+			    exp->e_reexport == REEXP_AUTO_FSIDNUM) == 0) {
+				errno = EINVAL;
+				return -1;
+			}
+			fsidnum = search_fsidnum;
+			qword_addint(&bp, &blen, exp->e_flags | NFSEXP_FSID);
+		} else {
+			qword_addint(&bp, &blen, exp->e_flags & flag_mask);
+		}
+
 		qword_addint(&bp, &blen, exp->e_anonuid);
 		qword_addint(&bp, &blen, exp->e_anongid);
-		qword_addint(&bp, &blen, exp->e_fsid);
+		qword_addint(&bp, &blen, fsidnum);
 
 #ifdef HAVE_JUNCTION_SUPPORT
 		write_fsloc(&bp, &blen, exp);
 #endif
-		write_secinfo(&bp, &blen, exp, flag_mask);
+		write_secinfo(&bp, &blen, exp, flag_mask, do_fsidnum ? NFSEXP_FSID : 0);
 		if (exp->e_uuid == NULL || different_fs) {
 			char u[16];
 			if ((exp->e_flags & flag_mask & NFSEXP_FSID) == 0 &&
Signature · 1 line
2.31.1
#4 lore raw

PATCH v3 4/8 export: Uncover NFS subvolume after reboot

In reply to Richard Weinberger

When a re-exporting NFS server reboots, none of the subvolumes are present. This is because the NFS client code will mount only upon first access. So, when we see an NFS handle with an yet unknown fsidnum, lookup in the reexport database for it.
If one is found, stat the path to trigger the mount.
That way stale NFS handles are avoided after a reboot.

---
 support/export/cache.c | 6 ++++++
 1 file changed, 6 insertions(+)
support/export/cache.c +6 -0
index 42a694d0..19bbba55 100644
@@ -778,6 +778,7 @@ static void nfsd_fh(int f)
 	int dev_missing = 0;
 	char buf[RPC_CHAN_BUF_SIZE], *bp;
 	int blen;
+	int did_uncover = 0;
 
 	blen = cache_read(f, buf, sizeof(buf));
 	if (blen <= 0 || buf[blen-1] != '\n') return;
@@ -815,6 +816,11 @@ static void nfsd_fh(int f)
 		for (exp = exportlist[i].p_head; exp; exp = next_exp) {
 			char *path;
 
+			if (!did_uncover && parsed.fsidnum && parsed.fsidtype == FSID_NUM && exp->m_export.e_reexport != REEXP_NONE) {
+				reexpdb_uncover_subvolume(parsed.fsidnum);
+				did_uncover = 1;
+			}
+
 			if (exp->m_export.e_flags & NFSEXP_CROSSMOUNT) {
 				static nfs_export *prev = NULL;
 				static void *mnt = NULL;
Signature · 1 line
2.31.1
#5 lore raw

PATCH v3 5/8 exports.man: Document reexport= option

In reply to Richard Weinberger
---
 utils/exportfs/exports.man | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
utils/exportfs/exports.man +31 -0
index 83dd6807..b7582776 100644
@@ -468,6 +468,37 @@ will only work if all clients use a consistent security policy.  Note
 that early kernels did not support this export option, and instead
 enabled security labels by default.
 
+.TP
+.IR reexport= auto-fsidnum|predefined-fsidnum
+This option helps when a NFS share is re-exported. Since the NFS server
+needs a unique identifier for each exported filesystem and a NFS share
+cannot provide such, usually a manual fsid is needed.
+As soon
+.IR crossmnt
+is used manually assigning fsid won't work anymore. This is where this
+option becomes handy. It will automatically assign a numerical fsid
+to exported NFS shares. The fsid and path relations are stored in a SQLite
+database. If
+.IR auto-fsidnum
+is selected, the fsid is also autmatically allocated.
+.IR predefined-fsidnum
+assumes pre-allocated fsid numbers and will just look them up.
+This option depends also on the kernel, you will need at least kernel version
+5.19.
+Since
+.IR reexport=
+can automatically allocate and assign numerical fsids, it is no longer possible
+to have numerical fsids in other exports as soon this option is used in at least
+one export entry.
+
+The association between fsid numbers and paths is stored in a SQLite database.
+Don't edit or remove the database unless you know exactly what you're doing.
+.IR predefined-fsidnum
+is useful when you have used
+.IR auto-fsidnum
+before and don't want further entries stored.
+
+
 .SS User ID Mapping
 .PP
 .B nfsd
Signature · 1 line
2.31.1
#6 lore raw

PATCH v3 6/8 reexport: Add sqlite backend

In reply to Richard Weinberger

The reexport database code is designed to support multiple ways to store the state.
So far only SQlite is implemented.

---
 support/reexport/backend_sqlite.c   | 267 ++++++++++++++++++++++++++++
 support/reexport/reexport_backend.h |  47 +++++
 2 files changed, 314 insertions(+)
 create mode 100644 support/reexport/backend_sqlite.c
 create mode 100644 support/reexport/reexport_backend.h
support/reexport/backend_sqlite.c new file +267 -0
new file mode 100644
index 00000000..132f30c4
@@ -0,0 +1,267 @@
+#ifdef HAVE_CONFIG_H
+#include <config.h>
+#endif
+
+#include <sqlite3.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/random.h>
+#include <unistd.h>
+
+#include "conffile.h"
+#include "reexport_backend.h"
+#include "xlog.h"
+
+#define REEXPDB_DBFILE NFS_STATEDIR "/reexpdb.sqlite3"
+#define REEXPDB_DBFILE_WAIT_USEC (5000)
+
+static sqlite3 *db;
+static int init_done;
+
+static int prng_init(void)
+{
+	int seed;
+
+	if (getrandom(&seed, sizeof(seed), 0) != sizeof(seed)) {
+		xlog(L_ERROR, "Unable to obtain seed for PRNG via getrandom()");
+		return -1;
+	}
+
+	srand(seed);
+	return 0;
+}
+
+static void wait_for_dbaccess(void)
+{
+	usleep(REEXPDB_DBFILE_WAIT_USEC + (rand() % REEXPDB_DBFILE_WAIT_USEC));
+}
+
+static bool sqlite_plug_init(void)
+{
+	char *sqlerr;
+	int ret;
+
+	if (init_done)
+		return true;
+
+	if (prng_init() != 0)
+		return false;
+
+	ret = sqlite3_open_v2(conf_get_str_with_def("reexport", "sqlitedb", REEXPDB_DBFILE),
+			      &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX,
+			      NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to open reexport database: %s", sqlite3_errstr(ret));
+		return false;
+	}
+
+again:
+	ret = sqlite3_exec(db, "CREATE TABLE IF NOT EXISTS fsidnums (num INTEGER PRIMARY KEY CHECK (num > 0 AND num < 4294967296), path TEXT UNIQUE); CREATE INDEX IF NOT EXISTS idx_ids_path ON fsidnums (path);", NULL, NULL, &sqlerr);
+	switch (ret) {
+	case SQLITE_OK:
+		init_done = 1;
+		ret = 0;
+		break;
+	case SQLITE_BUSY:
+	case SQLITE_LOCKED:
+		wait_for_dbaccess();
+		goto again;
+	default:
+		xlog(L_ERROR, "Unable to init reexport database: %s", sqlite3_errstr(ret));
+		sqlite3_free(sqlerr);
+		sqlite3_close_v2(db);
+		ret = -1;
+	}
+
+	return ret == 0 ? true : false;
+}
+
+static void sqlite_plug_destroy(void)
+{
+	if (!init_done)
+		return;
+
+	sqlite3_close_v2(db);
+}
+
+static bool get_fsidnum_by_path(char *path, uint32_t *fsidnum, bool *found)
+{
+	static const char fsidnum_by_path_sql[] = "SELECT num FROM fsidnums WHERE path = ?1;";
+	sqlite3_stmt *stmt = NULL;
+	bool success = false;
+	int ret;
+
+	*found = false;
+
+	ret = sqlite3_prepare_v2(db, fsidnum_by_path_sql, sizeof(fsidnum_by_path_sql), &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to prepare SQL query '%s': %s", fsidnum_by_path_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+	ret = sqlite3_bind_text(stmt, 1, path, -1, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to bind SQL query '%s': %s", fsidnum_by_path_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+again:
+	ret = sqlite3_step(stmt);
+	switch (ret) {
+	case SQLITE_ROW:
+		*fsidnum = sqlite3_column_int(stmt, 0);
+		success = true;
+		*found = true;
+		break;
+	case SQLITE_DONE:
+		/* No hit */
+		success = true;
+		*found = false;
+		break;
+	case SQLITE_BUSY:
+	case SQLITE_LOCKED:
+		wait_for_dbaccess();
+		goto again;
+	default:
+		xlog(L_WARNING, "Error while looking up '%s' in database: %s", path, sqlite3_errstr(ret));
+	}
+
+out:
+	sqlite3_finalize(stmt);
+	return success;
+}
+
+static bool sqlite_plug_path_by_fsidnum(uint32_t fsidnum, char **path, bool *found)
+{
+	static const char path_by_fsidnum_sql[] = "SELECT path FROM fsidnums WHERE num = ?1;";
+	sqlite3_stmt *stmt = NULL;
+	bool success = false;
+	int ret;
+
+	*found = false;
+
+	ret = sqlite3_prepare_v2(db, path_by_fsidnum_sql, sizeof(path_by_fsidnum_sql), &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to prepare SQL query '%s': %s", path_by_fsidnum_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+	ret = sqlite3_bind_int(stmt, 1, fsidnum);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to bind SQL query '%s': %s", path_by_fsidnum_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+again:
+	ret = sqlite3_step(stmt);
+	switch (ret) {
+	case SQLITE_ROW:
+		*path = strdup((char *)sqlite3_column_text(stmt, 0));
+		if (*path) {
+			*found = true;
+			success = true;
+		} else {
+			xlog(L_WARNING, "Out of memory");
+		}
+		break;
+	case SQLITE_DONE:
+		/* No hit */
+		*found = false;
+		success = true;
+		break;
+	case SQLITE_BUSY:
+	case SQLITE_LOCKED:
+		wait_for_dbaccess();
+		goto again;
+	default:
+		xlog(L_WARNING, "Error while looking up '%i' in database: %s", fsidnum, sqlite3_errstr(ret));
+	}
+
+out:
+	sqlite3_finalize(stmt);
+	return success;
+}
+
+static bool new_fsidnum_by_path(char *path, uint32_t *fsidnum)
+{
+	/*
+	 * This query is a little tricky. We use SQL to find and claim the smallest free fsid number.
+	 * To find a free fsid the fsidnums is left joined to itself but with an offset of 1.
+	 * Everything after the UNION statement is to handle the corner case where fsidnums
+	 * is empty. In this case we want 1 as first fsid number.
+	 */
+	static const char new_fsidnum_by_path_sql[] = "INSERT INTO fsidnums VALUES ((SELECT ids1.num + 1 FROM fsidnums AS ids1 LEFT JOIN fsidnums AS ids2 ON ids2.num = ids1.num + 1 WHERE ids2.num IS NULL UNION SELECT 1 WHERE NOT EXISTS (SELECT NULL FROM fsidnums WHERE num = 1) LIMIT 1), ?1) RETURNING num;";
+
+	sqlite3_stmt *stmt = NULL;
+	int ret, check = 0;
+	bool success = false;
+
+	ret = sqlite3_prepare_v2(db, new_fsidnum_by_path_sql, sizeof(new_fsidnum_by_path_sql), &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to prepare SQL query '%s': %s", new_fsidnum_by_path_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+	ret = sqlite3_bind_text(stmt, 1, path, -1, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_WARNING, "Unable to bind SQL query '%s': %s", new_fsidnum_by_path_sql, sqlite3_errstr(ret));
+		goto out;
+	}
+
+again:
+	ret = sqlite3_step(stmt);
+	switch (ret) {
+	case SQLITE_ROW:
+		*fsidnum = sqlite3_column_int(stmt, 0);
+		success = true;
+		break;
+	case SQLITE_CONSTRAINT:
+		/* Maybe we lost the race against another writer and the path is now present. */
+		check = 1;
+		break;
+	case SQLITE_BUSY:
+	case SQLITE_LOCKED:
+		wait_for_dbaccess();
+		goto again;
+	default:
+		xlog(L_WARNING, "Error while looking up '%s' in database: %s", path, sqlite3_errstr(ret));
+	}
+
+out:
+	sqlite3_finalize(stmt);
+
+	if (check) {
+		bool found = false;
+
+		get_fsidnum_by_path(path, fsidnum, &found);
+		if (!found)
+			xlog(L_WARNING, "SQLITE_CONSTRAINT error while inserting '%s' in database", path);
+	}
+
+	return success;
+}
+
+static bool sqlite_plug_fsidnum_by_path(char *path, uint32_t *fsidnum, int may_create, bool *found)
+{
+	bool success;
+
+	success = get_fsidnum_by_path(path, fsidnum, found);
+	if (success) {
+		if (!*found && may_create) {
+			success = new_fsidnum_by_path(path, fsidnum);
+			if (success)
+				*found = true;
+		}
+	}
+
+	return success;
+}
+
+struct reexpdb_backend_plugin sqlite_plug_ops = {
+	.fsidnum_by_path = sqlite_plug_fsidnum_by_path,
+	.path_by_fsidnum = sqlite_plug_path_by_fsidnum,
+	.initdb = sqlite_plug_init,
+	.destroydb = sqlite_plug_destroy,
+};
support/reexport/reexport_backend.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.31.1
#7 lore raw

PATCH v3 7/8 export: Add fsidd

In reply to Richard Weinberger

The fsidnum daemon offers a local UNIX domain socket interface for all NFS userspace to query the reexport database. Currently fsidd just uses the SQlite backend.

fsidd serves also as an example on how to implement more complex backends for the load balancing use case.

---
 support/reexport/Makefile.am |  12 +++
 support/reexport/fsidd.c     | 198 +++++++++++++++++++++++++++++++++++
 2 files changed, 210 insertions(+)
 create mode 100644 support/reexport/fsidd.c
support/reexport/Makefile.am +12 -0
index 9d544a8f..fbd66a20 100644
@@ -3,4 +3,16 @@
 noinst_LIBRARIES = libreexport.a
 libreexport_a_SOURCES = reexport.c
 
+sbin_PROGRAMS	= fsidd
+
+fsidd_SOURCES = fsidd.c backend_sqlite.c
+
+fsidd_LDADD = ../../support/misc/libmisc.a \
+	      ../../support/nfs/libnfs.la \
+	       $(LIBPTHREAD) $(LIBEVENT) $(LIBSQLITE) \
+	       $(OPTLIBS)
+
+fsidd_CPPFLAGS = $(AM_CPPFLAGS) $(CPPFLAGS) \
+		  -I$(top_builddir)/support/include
+
 MAINTAINERCLEANFILES = Makefile.in
support/reexport/fsidd.c new file +198 -0

198 lines not on this page · raw message

Signature · 1 line
2.31.1
#8 lore raw

PATCH v3 8/8 Add fsid systemd service file

In reply to Richard Weinberger
---
 systemd/Makefile.am   |  3 ++-
 systemd/fsidd.service | 10 ++++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 systemd/fsidd.service
systemd/Makefile.am +2 -1
index 2e250dca..b4483222 100644
@@ -15,7 +15,8 @@ unit_files =  \
     rpc-statd-notify.service \
     rpc-statd.service \
     \
-    proc-fs-nfsd.mount
+    proc-fs-nfsd.mount \
+    fsidd.service
 
 rpc_pipefs_mount_file = \
     var-lib-nfs-rpc_pipefs.mount
systemd/fsidd.service new file +10 -0
new file mode 100644
index 00000000..9cb480e3
@@ -0,0 +1,10 @@
+[Unit]
+Description=NFS FSID Daemon
+After=local-fs.target
+Before=nfs-mountd.service nfs-server.service
+
+[Service]
+ExecStart=/usr/sbin/fsidd
+
+[Install]
+RequiredBy=nfs-mountd.service nfs-server.service
Signature · 1 line
2.31.1
#9 lore raw
In reply to Richard Weinberger
Quoted 76 lines · Richard Weinberger

After a longer hiatus I'm sending the next iteration of my re-export improvement patch series. While the kernel side is upstream since v6.2, the nfs-utils parts are still missing.
This patch series aims to solve this.

The core idea is adding new export option, reeport=
Using reexport= it is possible to mark an export entry in the exports file explicitly as NFS re-export and select a strategy on how unique identifiers should be provided. This makes the crossmnt feature work in the re-export case.
Currently two strategies are supported, "auto-fsidnum" and "predefined-fsidnum".

In my earlier series a sqlite database was mandatory to keep track of generated fsids.
This series follows a different approach, instead of directly using sqlite in all nfs-utils components (linking libsqlite), a new deamon manages the database, fsidd.
fsidd offers a simple (but stupid?) text based interface over a unix domain socket which can be queried by mountd, exportfs, etc. for fsidnums. The main idea behind fsidd is allowing users to implement their own fsidd which keeps global state across load balancers.
I'm still not happy with fsidd, there is room for improvement but first I'd like to know whether you like or hate this approach.

A typical export entry on a re-exporting server looks like:
/nfs *(rw,no_root_squash,no_subtree_check,crossmnt,reexport=auto-fsidnum)
reexport=auto-fsidnum will automatically assign an fsid= to /nfs and all uncovered subvolumes.

Changes since v2, https://lore.kernel.org/linux-nfs/[email protected]/
- Split patch series
- Add improved fsidd system unit file
- Rebased to nfs-utils master as of today
- Dropped init code from exportd

Changes since v1, https://lore.kernel.org/linux-nfs/[email protected]/
- Factor out Sqlite and put it into a daemon
- Add fsidd
- Basically re-implemented the patch series
- Lot's of fixes (e.g. nfs v4 root export)

Richard Weinberger (8):
Add reexport helper library
Implement reexport= export option
export: Wireup reexport mechanism
export: Uncover NFS subvolume after reboot
exports.man: Document reexport= option
reexport: Add sqlite backend
export: Add fsidd
Add fsid systemd service file

  configure.ac                        |   1 +
  support/Makefile.am                 |   2 +-
  support/export/Makefile.am          |   2 +
  support/export/cache.c              |  74 ++++++-
  support/export/export.c             |  20 ++
  support/include/nfslib.h            |   1 +
  support/nfs/Makefile.am             |   1 +
  support/nfs/exports.c               |  62 ++++++
  support/reexport/Makefile.am        |  18 ++
  support/reexport/backend_sqlite.c   | 267 +++++++++++++++++++++++
  support/reexport/fsidd.c            | 198 +++++++++++++++++
  support/reexport/reexport.c         | 326 ++++++++++++++++++++++++++++
  support/reexport/reexport.h         |  18 ++
  support/reexport/reexport_backend.h |  47 ++++
  systemd/Makefile.am                 |   5 +-
  systemd/fsidd.service               |  10 +
  utils/exportd/Makefile.am           |   4 +-
  utils/exportfs/Makefile.am          |   3 +
  utils/exportfs/exportfs.c           |  11 +
  utils/exportfs/exports.man          |  31 +++
  utils/mount/Makefile.am             |   3 +-
  utils/mountd/Makefile.am            |   2 +
  22 files changed, 1096 insertions(+), 10 deletions(-)
Quoted 8 lines

create mode 100644 support/reexport/Makefile.am create mode 100644 support/reexport/backend_sqlite.c create mode 100644 support/reexport/fsidd.c create mode 100644 support/reexport/reexport.c create mode 100644 support/reexport/reexport.h create mode 100644 support/reexport/reexport_backend.h create mode 100644 systemd/fsidd.service

Committed... (tag: nfs-utils-2-6-3-rc9)

Thank you!

steved.