Skip to contentSkip to conversation

mountd: export: Deal with NFS filesystems

RFC PATCH In review Richard Weinberger 2 participants 5 messages Jan 31, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 mountd: export: Deal with NFS filesystems

When a NFS filesystem is exported usually NFSD refuses to export the filesystem because mountd was unable to provide an UUID and the kernel cannot derive a dev id from the NFS client super block.

To deal with this situation, teach uuid_by_path() how to generate an UUID from a NFS filesystem.

Using /proc/fs/nfsfs/volumes it is possible to find the NFS fsid from the backend and use it as seed for mountd's UUID mechanism.

---
Hello,

This is the userspace side to make cross mounts work for NFS re-exports. The basic idea is that mountd takes the NFS fsid from the backend server and feeds into mountd'd existing UUID mechanism.

Getting the NFS fsid is currently a bit clumsy, mountd has to scan /proc/fs/nfsfs/volumes. With the upcoming fsinfo() system call this information could be transferred in a much simpler way. e.g. NFS client stores the fsid in its sb->s_uuid and fsinfo() exposes sb->s_uuid to userspace.

This approach works for v3-to-v4 and v4-to-v4 re-exports. v3-to-v3 does *not* work, the reason is that nfs_encode_fh() aborts due to not enough space in the fhandle.
I'm still investigating on how to improve this case.

Currently I’m toying around with a NFS fsid to export FSID_NUM mappings scheme which are stored to disk.
Another possibility is a new fsid type for NFSD, where the raw NFS fsid from the backend server is mirrored. Although such a mirror mode makes local exports impossible.
At the end of the day this is something the admin has to decide. I don't think there will be a generic solution which satisfies all needs.

Looking forward for your feedback. :-)

Thanks,
//richard

---
 support/export/cache.c | 65 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 61 insertions(+), 4 deletions(-)
support/export/cache.c +61 -4
index a5823e92e9f2..053ad86619b8 100644
@@ -331,6 +331,51 @@ static const unsigned long nonblkid_filesystems[] = {
     0        /* last */
 };
 
+static int get_uuid_from_fsid(char *path, char *uuid_str, size_t len)
+{
+	unsigned int min_dev, maj_dev, min_fsid, maj_fsid;
+	int rc, n, found = 0, header_seen = 0;
+	struct stat stb;
+	FILE *nfsfs_fd;
+	char line[128];
+
+	rc = nfsd_path_stat(path, &stb);
+	if (rc) {
+		xlog(L_WARNING, "Unable to stat %s", path);
+		return 0;
+	}
+
+	nfsfs_fd = fopen("/proc/fs/nfsfs/volumes", "r");
+	if (nfsfs_fd == NULL) {
+		xlog(L_WARNING, "Unable to open nfsfs volume file: %m");
+		return 0;
+	}
+
+	while (fgets(line, sizeof(line), nfsfs_fd) != NULL) {
+		if (!header_seen) {
+			header_seen = 1;
+			continue;
+		}
+		n = sscanf(line, "v%*u %*x %*u %u:%u %x:%x %*s", &maj_dev,
+			   &min_dev, &maj_fsid, &min_fsid);
+
+		if (n != 4) {
+			xlog(L_WARNING, "Unable to parse nfsfs volume line: %d, %s", n, line);
+			continue;
+		}
+
+		if (makedev(maj_dev, min_dev) == stb.st_dev) {
+			found = 1;
+			snprintf(uuid_str, len, "%08x%08x", maj_fsid, min_fsid);
+			break;
+		}
+	}
+
+	fclose(nfsfs_fd);
+
+	return found;
+}
+
 static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 {
 	/* get a uuid for the filesystem found at 'path'.
@@ -362,7 +407,7 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 	 */
 	struct statfs64 st;
 	char fsid_val[17];
-	const char *blkid_val = NULL;
+	const char *fsuuid_val = NULL;
 	const char *val;
 	int rc;
 
@@ -375,7 +420,19 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 				break;
 		}
 		if (*bad == 0)
-			blkid_val = get_uuid_blkdev(path);
+			fsuuid_val = get_uuid_blkdev(path);
+		else if (*bad == 0x6969 /* NFS_SUPER_MAGIC */) {
+			char tmp[17];
+			int ret = get_uuid_from_fsid(path, tmp, sizeof(tmp));
+
+			if (ret < 0) {
+				xlog(L_WARNING, "Unable to read nfsfs volume file: %i", ret);
+			} else if (ret == 0) {
+				xlog(L_WARNING, "Unable to find nfsfs volume entry for %s", path);
+			} else {
+				fsuuid_val = tmp;
+			}
+		}
 	}
 
 	if (rc == 0 &&
@@ -385,8 +442,8 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 	else
 		fsid_val[0] = 0;
 
-	if (blkid_val && (type--) == 0)
-		val = blkid_val;
+	if (fsuuid_val && (type--) == 0)
+		val = fsuuid_val;
 	else if (fsid_val[0] && (type--) == 0)
 		val = fsid_val;
 	else
Signature · 1 line
2.26.2
#1 lore raw

Thanks for continuing to look into this, it's interesting work.

Quoted 10 lines · Richard Weinberger

When a NFS filesystem is exported usually NFSD refuses to export the filesystem because mountd was unable to provide an UUID and the kernel cannot derive a dev id from the NFS client super block.

To deal with this situation, teach uuid_by_path() how to generate an UUID from a NFS filesystem.

Using /proc/fs/nfsfs/volumes it is possible to find the NFS fsid from the backend and use it as seed for mountd's UUID mechanism.

Sorry, I haven't checked--what is that number, really? It's probably not the fsid returned from the server, as that wouldn't be guaranteed unique across multiple servers. But that means it's probably generated in a way that doesn't guarantee it's stable across reboots. And we need filehandles to work across reboots.

I think we've got to generate fsid's ourself and keep an on-disk database.

Quoted 10 lines

This is the userspace side to make cross mounts work for NFS re-exports. The basic idea is that mountd takes the NFS fsid from the backend server and feeds into mountd'd existing UUID mechanism.

Getting the NFS fsid is currently a bit clumsy, mountd has to scan /proc/fs/nfsfs/volumes. With the upcoming fsinfo() system call this information could be transferred in a much simpler way. e.g. NFS client stores the fsid in its sb->s_uuid and fsinfo() exposes sb->s_uuid to userspace.

Quoted 4 lines

This approach works for v3-to-v4 and v4-to-v4 re-exports. v3-to-v3 does *not* work, the reason is that nfs_encode_fh() aborts due to not enough space in the fhandle.
I'm still investigating on how to improve this case.

We might be able to do better, but I think a v4-only solution to start would be OK.

Quoted 2 lines

Currently I’m toying around with a NFS fsid to export FSID_NUM mappings scheme which are stored to disk.

I think something like that's more like what we need.

Quoted 5 lines

Another possibility is a new fsid type for NFSD, where the raw NFS fsid from the backend server is mirrored. Although such a mirror mode makes local exports impossible.
At the end of the day this is something the admin has to decide. I don't think there will be a generic solution which satisfies all needs.

We can provide options if we need to, but the costs of options can be underestimated. Ideally we'd at least have a default that's safe and works for most cases.

--b.

Quoted 108 lines

Looking forward for your feedback. :-)

Thanks,
//richard

---
 support/export/cache.c | 65 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 61 insertions(+), 4 deletions(-)
support/export/cache.c +61 -4
index a5823e92e9f2..053ad86619b8 100644
@@ -331,6 +331,51 @@ static const unsigned long nonblkid_filesystems[] = {
     0        /* last */
 };
 
+static int get_uuid_from_fsid(char *path, char *uuid_str, size_t len)
+{
+	unsigned int min_dev, maj_dev, min_fsid, maj_fsid;
+	int rc, n, found = 0, header_seen = 0;
+	struct stat stb;
+	FILE *nfsfs_fd;
+	char line[128];
+
+	rc = nfsd_path_stat(path, &stb);
+	if (rc) {
+		xlog(L_WARNING, "Unable to stat %s", path);
+		return 0;
+	}
+
+	nfsfs_fd = fopen("/proc/fs/nfsfs/volumes", "r");
+	if (nfsfs_fd == NULL) {
+		xlog(L_WARNING, "Unable to open nfsfs volume file: %m");
+		return 0;
+	}
+
+	while (fgets(line, sizeof(line), nfsfs_fd) != NULL) {
+		if (!header_seen) {
+			header_seen = 1;
+			continue;
+		}
+		n = sscanf(line, "v%*u %*x %*u %u:%u %x:%x %*s", &maj_dev,
+			   &min_dev, &maj_fsid, &min_fsid);
+
+		if (n != 4) {
+			xlog(L_WARNING, "Unable to parse nfsfs volume line: %d, %s", n, line);
+			continue;
+		}
+
+		if (makedev(maj_dev, min_dev) == stb.st_dev) {
+			found = 1;
+			snprintf(uuid_str, len, "%08x%08x", maj_fsid, min_fsid);
+			break;
+		}
+	}
+
+	fclose(nfsfs_fd);
+
+	return found;
+}
+
 static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 {
 	/* get a uuid for the filesystem found at 'path'.
@@ -362,7 +407,7 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 	 */
 	struct statfs64 st;
 	char fsid_val[17];
-	const char *blkid_val = NULL;
+	const char *fsuuid_val = NULL;
 	const char *val;
 	int rc;
 
@@ -375,7 +420,19 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 				break;
 		}
 		if (*bad == 0)
-			blkid_val = get_uuid_blkdev(path);
+			fsuuid_val = get_uuid_blkdev(path);
+		else if (*bad == 0x6969 /* NFS_SUPER_MAGIC */) {
+			char tmp[17];
+			int ret = get_uuid_from_fsid(path, tmp, sizeof(tmp));
+
+			if (ret < 0) {
+				xlog(L_WARNING, "Unable to read nfsfs volume file: %i", ret);
+			} else if (ret == 0) {
+				xlog(L_WARNING, "Unable to find nfsfs volume entry for %s", path);
+			} else {
+				fsuuid_val = tmp;
+			}
+		}
 	}
 
 	if (rc == 0 &&
@@ -385,8 +442,8 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid)
 	else
 		fsid_val[0] = 0;
 
-	if (blkid_val && (type--) == 0)
-		val = blkid_val;
+	if (fsuuid_val && (type--) == 0)
+		val = fsuuid_val;
 	else if (fsid_val[0] && (type--) == 0)
 		val = fsid_val;
 	else
Signature · 1 line
2.26.2
#2 lore raw

----- Ursprüngliche Mail -----

Quoted 9 lines

Von: "bfields" <[email protected]>

Quoted 2 lines

Using /proc/fs/nfsfs/volumes it is possible to find the NFS fsid from the backend and use it as seed for mountd's UUID mechanism.

Sorry, I haven't checked--what is that number, really? It's probably not the fsid returned from the server, as that wouldn't be guaranteed unique across multiple servers. But that means it's probably generated in a way that doesn't guarantee it's stable across reboots. And we need filehandles to work across reboots.

Unless I badly misunderstood the code it comes from fs/nfs/client.c's nfs_create_server() where the NFS client fetches NFS_ATTR_FATTR via getattr(). So it should be unique:
https://datatracker.ietf.org/doc/html/rfc7530#section-5.8.1.9

Thanks,
//richard

#3 lore raw
Quoted 11 lines · Richard Weinberger

----- Ursprüngliche Mail -----

Quoted 9 lines

Von: "bfields" <[email protected]>

Quoted 2 lines

Using /proc/fs/nfsfs/volumes it is possible to find the NFS fsid from the backend and use it as seed for mountd's UUID mechanism.

Sorry, I haven't checked--what is that number, really? It's probably not the fsid returned from the server, as that wouldn't be guaranteed unique across multiple servers. But that means it's probably generated in a way that doesn't guarantee it's stable across reboots. And we need filehandles to work across reboots.

Quoted 4 lines

Unless I badly misunderstood the code it comes from fs/nfs/client.c's nfs_create_server() where the NFS client fetches NFS_ATTR_FATTR via getattr(). So it should be unique:
https://datatracker.ietf.org/doc/html/rfc7530#section-5.8.1.9

Unique within one server, but not across different servers.

Depending on how your servers assign fsids, collisions might be quite likely. Results of a collision would be pretty bad.

--b.

#4 lore raw

----- Ursprüngliche Mail -----

Quoted 10 lines

Von: "bfields" <[email protected]>

Quoted 4 lines

Unless I badly misunderstood the code it comes from fs/nfs/client.c's nfs_create_server() where the NFS client fetches NFS_ATTR_FATTR via getattr(). So it should be unique:
https://datatracker.ietf.org/doc/html/rfc7530#section-5.8.1.9

Unique within one server, but not across different servers.

Depending on how your servers assign fsids, collisions might be quite likely. Results of a collision would be pretty bad.

Hmm. So this means when you have a setup where the same file system is exported via multiple servers (the load balancing use case), getattr on the same file can report a different fsid at any time?
This renders the fsid attribute close to useless :-(

Anyway, this is not a show stopper just makes more clear that we should take the FSID_NUM mapping path.
Please stay tuned, I'll send ASAP another patch.

Thanks,
//richard