Skip to contentSkip to conversation

nfsd vurlerability submit

PATCH In review J. Bruce Fields 9 participants 30 messages Jan 11, 2021 – Jan 22, 2021 18 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CAHxDmpTKJfnhGY9CVupyVYhNCTDVKBB6KRwh-E6u_XEPJq4WJQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): X/[email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CAHxDmpSp1LHzKD5uqbfi+jcnb+nFaAZbc5++E0oOvLsYvyYDpw@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CAHxDmpSjwrcr_fqLJa5=Zo=xmbt2Eo9dcy6TQuoU8+F3yVVNhw@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CAHxDmpR8S7NR8OU2nWJmWBdFU9a7wDuDnxviQ2E9RDOeW9fExg@mail.gmail.com

#10 lore raw
Quoted 5 lines · 吴异

Yes, the safest way for users is that the first exported file system is on a different hard disk with "\", and then does not contain subdirectories, so I seem to have nothing to do. However, I did not find these tips in the configuration files related and man exports . It is best to recommend the best configuration method to users if it is updated in the future.

I agree that the documentation needs improvement.

I also think we should warn by default when a user tries to export something with no_subtree_check that's not at a mountpoint.

(Really, "at a mountpoint" should be "at the root of a filesystem", but I don't know how to check for that, at least not from userspace. I'm not sure whether the warning belongs in the kernel or elsewhere.)

Quoted 4 lines

Back to the present situation, have you considered removing the following piece of code in the nfsd_acceptable function:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;

That would turn on subtree_check unconditionally. Currently we default to no_subtree_check, and for good reason: subtree_check causes unexpected behavior, like ESTALE on attempts to access open files after cross-directory rename.

Quoted 7 lines

In this way, all problems seem to be solved. Even if an attacker can get or guess some file handles, these handles will not actually work.Fix compose_entry_fh and nfsd_lookup_dentry functions is actually not that urgent.

One more question, do I need to copy all the previous emails to [email protected], this is a bit too much, right?

I don't think that's necessary. To summarize issue you've raised; if a client exports only a subdirectory of a filesystem:

- NFSv2/v3 clients can discover the filehandle of the parent of the export point using readdir_plus. I agree that we should fix this; I'll follow up with a patch. (It's already not too hard to guess filehandles, though.)

- Once a single filehandle outside the export directory is found, it's easy to access the rest of the filesystem. This issue has been known for decades. But I don't think it's well-enough understood by users.

- If only a subdirectory of a filesystem is exported, an attacker might be able to gain access to *other* filesystems as well, by replacing some component along the exported path by a symlink and waiting for the server to restart. (In the v2/v3 case, export path symlinks are followed on the server side.)

The main defense against all of this is to *only* export the root of a filesystem. If you're exporting "/export/home/", then you should create a separate filesystem and mount that at "/export/home/".

But this isn't as well understood as it should be, and we should do a better job of preventing users from choosing dangerous configurations. And then mitigate the danger where we can.

--b.

Quoted 196 lines

Best regards,

Quoted 191 lines · J. Bruce Fields <[email protected]> 于2021年1月11日周一 上午4:28写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异

I do try it,because of the default no_subtreecheck in nfsd, I can

delete

Quoted 1 line
Quoted 1 line

/nfs as long as I can get the handle of /fs. This takes advantage of

the

Quoted 9 lines
Quoted 9 lines

vulnerability I submitted before. I have added my poc and traffic information in the attachment.

My server filesystem configuration is as follows:

/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro)

/dev/sda1 on /fs type ext4 (rw,relatime)

Quoted 9 lines
Quoted 3 lines

The configuration of /etc/export is

/fs/nfs *(rw,sync,no_root_squash,insecure)

Again, you're exporting a subdirectory, which we discourage. The safe and recommended configuration would be to mount /dev/sda1 on /fs/nfs and export /fs/nfs.

Quoted 1 line

As you can see, the file system where the shared directory is located

has

Quoted 1 line
Quoted 1 line

been separated from the “\” directory.What I did was to delete the

mount

Quoted 4 lines
Quoted 1 line

point directory

It should not be possible to delete /fs/nfs/ while you had something mounted there.

Whoops, I'm sorry, there's only a mountpoint at /fs/, so, yes, you can probably delete /fs/nfs and replace it by a symlink, got it.

Sure, that's interesting. But, again, we don't recommend exporting filesystem subdirectories.

Good to keep the public nfs list ([email protected]) cc'd on these things, by the way.

--b.

Quoted 5 lines

--b.

Quoted 2 lines

and restart the server-side nfsd service. At this time, the server-side shared directory became "/", and the configuration

information

Quoted 8 lines
Quoted 8 lines

did not change.

For the convenience I set no_root_squash on the server, but in fact, as long as the server gives the user the permission to write in the /fs directory, the above operation can be completed.

Quoted 8 lines
Quoted 8 lines

Best regards,

Quoted 2 lines · J. Bruce Fields <[email protected]> 于2021年1月9日周六 上午12:44写道:
Quoted 1 line · 吴异

when the shared directory /fs/nfs of the server is mounted on the

device

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/dev/sda3, and the "/" directory is mounted on the device

/dev/sda1, then

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the file system of the shared directory and the "/" file system

are In

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

isolation, an attacker can obtain the ability of any file in the

/dev/sda3

Quoted 1 line

file system by using a file handle, but cannot access the "/" file

system

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

by using an NFS file handle, even if he already knows the "/"

handle is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

"0100010000000000", which will return the error code nfserr_stale.

In this case, the attacker deletes the mount point directory

/fs/nfs in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the file system under its control., and create a symbolic link

with the

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

same name pointing to the "/" directory.

Access to the filesystem mounted at /fs/nfs/ should not give you the ability to delete the directory /fs/nfs.

Have you actually tried this?

Please explain what access you assume the attacker starts with, what they're able to do it, why you think that isn't expected, and please test to make sure your idea works.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

--b.

Quoted 2 lines

After nfsd re-parses the
configuration file /etc/exports, the attacker can remount it to

achieve

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

real file system escape.

It seems that nfsd should strengthen the restrictions on symbolic

link

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

resolution in configuration files.

Best regards,

Quoted 2 lines · J. Bruce Fields <[email protected]> 于2021年1月8日周五 下午11:20写道:
Quoted 1 line · Greg KH

On Tue, Jan 05, 2021 at 11:56:33AM -0500, J. Bruce Fields

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · 吴异
Quoted 2 lines

It is not safe to export only the root of the file system, Attacker can use lookup ".." to get the file handle to the

".."

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

directory,

Quoted 2 lines
Quoted 2 lines

You have claimed the ability to read and write any file on

the file

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

server. But if you put /fs/nfs/ on a separate filesystem

then

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

that is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

no longer possible. Perhaps you can learn the value of the

filehandle

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/fs/, but you can't use that filehandle to gain additional

access

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

unless

Quoted 3 lines
Quoted 3 lines

something on that filesystem is already exported.

So, there may be some information disclosure here, I agree,

but

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

it's

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

relatively mild, given that filehandles are not well-kept

secrets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

anyway.

Ok, to clarify, you are saying that this really isn't an issue

at all

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

with the kernel code, so there's nothing to do here?

Just trying to figure out if there's anything to actually work

on

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

here

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

or not.

I think there's one possibly interesting issue, but it doesn't

require

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

special handling. Reporter has raised a few issues:

- The one that might be interesting: clients may be able

to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

discover filehandles that they shouldn't be able to

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

readdir_plus. On review of the code, I agree it's odd

that

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the logic in compose_entry_fh() is different from that

in

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

nfsd_lookup_dentry(). I'd also prefer to take that

discussion

Quoted 1 line
Quoted 1 line

to [email protected], as this seems at most a

mild

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

information disclosure.

- If only part of a filesystem is exported, an attacker

can

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

gain

Quoted 1 line
Quoted 1 line

access to files outside the exported directory by

guessing

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

filehandles. This issue has been known for decades.

If we

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

want to discuss mitigation strategies, I believe the

correct

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

forum is again [email protected], not [email protected].

- An attacker might replace an exported mountpoint by a

symlink

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

pointing elsewhere. That seems to assume the attacker

already

Quoted 1 line
Quoted 1 line

has a lot of access, so I'm not clear what advantage

they

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

gain

Quoted 4 lines
Quoted 4 lines

from this.

--b.

#11 lore raw

PATCH 1/1 nfsd4: readdirplus shouldn't return parent of export

From: "J. Bruce Fields" <[email protected]>

If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.

The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.

Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.

However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.

---
 fs/nfsd/nfs3xdr.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/nfsd/nfs3xdr.c +2 -1
index 821db21ba072..34b880211e5e 100644
@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,
 	if (isdotent(name, namlen)) {
 		if (namlen == 2) {
 			dchild = dget_parent(dparent);
-			/* filesystem root - cannot return filehandle for ".." */
+			/*
+			 * Don't return filehandle for ".." if we're at

OA+ * the filesystem or export root:
+ */
if (dchild == dparent)
goto out;
+ if (dparent == exp->ex_path.dentry)
+ goto out;
} else
dchild = dget(dparent);
} else

Signature · 1 line
2.29.2
#12 lore raw

Re: [PATCH] nfsd4: readdirplus shouldn't return parent of export

Quoted 44 lines · J. Bruce Fields

From: "J. Bruce Fields" <[email protected]>

If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.

The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.

Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.

However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.

---
fs/nfsd/nfs3xdr.c | 7 ++++++-

1 file changed, 6 insertions(+), 1 deletion(-)
fs/nfsd/nfs3xdr.c +2 -1
index 821db21ba072..34b880211e5e 100644
@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,
 	if (isdotent(name, namlen)) {
 		if (namlen == 2) {
 			dchild = dget_parent(dparent);
-			/* filesystem root - cannot return filehandle for ".." */
+			/*
+			 * Don't return filehandle for ".." if we're at

OA+ * the filesystem or export root:

fs/nfsd/nfs3xdr.c +1 -0
+			 */
			if (dchild == dparent)
				goto out;
fs/nfsd/nfs3xdr.c +2 -0
+			if (dparent == exp->ex_path.dentry)
+				goto out;
Quoted 5 lines
		} else
			dchild = dget(dparent);
	} else
Signature · 1 line
2.29.2

Thanks for the fix!

I've replaced the Reported-by: tag and pushed this to my cel-next topic branch, and intend to submit it with the next 5.11 -rc pull request. See:

https://git.linux-nfs.org/?p=cel/cel-2.6.git;a=shortlog;h=refs/heads/cel-next

Is there additional context that should be added? A Link:
tag that points to the discussion on security@ perhaps?

Note there was some damage in the patch body: there's a spurious "OA" in the hunk that had to be removed before the patch would apply.

--
Chuck Lever

#13 lore raw

Re: [PATCH] nfsd4: readdirplus shouldn't return parent of export

Quoted 52 lines · Chuck Lever
Quoted 44 lines · J. Bruce Fields

From: "J. Bruce Fields" <[email protected]>

If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.

The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.

Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.

However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.

---
fs/nfsd/nfs3xdr.c | 7 ++++++-

1 file changed, 6 insertions(+), 1 deletion(-)
fs/nfsd/nfs3xdr.c +2 -1
index 821db21ba072..34b880211e5e 100644
@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,
 	if (isdotent(name, namlen)) {
 		if (namlen == 2) {
 			dchild = dget_parent(dparent);
-			/* filesystem root - cannot return filehandle for ".." */
+			/*
+			 * Don't return filehandle for ".." if we're at

OA+ * the filesystem or export root:

fs/nfsd/nfs3xdr.c +1 -0
+			 */
			if (dchild == dparent)
				goto out;
fs/nfsd/nfs3xdr.c +2 -0
+			if (dparent == exp->ex_path.dentry)
+				goto out;
Quoted 5 lines
		} else
			dchild = dget(dparent);
	} else
Signature · 1 line
2.29.2

Thanks for the fix!

Quoted 8 lines

I've replaced the Reported-by: tag and pushed this to my cel-next topic branch, and intend to submit it with the next 5.11 -rc pull request. See:

https://git.linux-nfs.org/?p=cel/cel-2.6.git;a=shortlog;h=refs/heads/cel-next

Is there additional context that should be added? A Link:
tag that points to the discussion on security@ perhaps?

I don't think so.

I guess it should get a stable cc: too.

Quoted 3 lines

Note there was some damage in the patch body: there's a spurious "OA" in the hunk that had to be removed before the patch would apply.

Whoops, apologies, I'm not sure how that happened....

--b.

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CAHxDmpR1zG25ADfK2jat4VKGbAOCg6YM_0WA+a_jQE82hbnMjA@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAHxDmpRfmVukMR_yF4coioiuzrsp72zBraHWZ8gaMydUuLwKFg@mail.gmail.com

#20 lore raw
Quoted 8 lines · 吴异

Telling users how to configure the exported file system in the most secure way does
mitigate the problem to some extent, but this does not seem to address the security risks posed by no_ subtree_ check in the code. In my opinion,when the generated filehandle does not contain the inode information of the parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find parent directories from any given file. (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)

Quoted 13 lines

I have a bold idea, why not directly remove the file handle modification in subtree_check, and then normalize the judgment of whether dentry exceeds the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by default is that it will cause problems when reading and writing files, rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system does not prohibit the export of a subdirectory).

Quoted 6 lines

Enabling subtree_check to add parent directoryinformation only brings some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and time-consuming change.

Another approach is to add more entropy to filehandles so they're harder to guess; see e.g.:

https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

#21 lore raw
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file.  (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.  We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#22 lore raw

I was under the impression that the best practice is to create something along the lines of

/srv/nfs

and then bind mount everything you plan to export into that folder; e.g.

/etc/fstab:
/data2/xray /srv/nfs/xray none defaults,bind 0

Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.

Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file.  (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.  We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

#23 lore raw
Quoted 9 lines · Patrick Goetz

I was under the impression that the best practice is to create something along the lines of

/srv/nfs

and then bind mount everything you plan to export into that folder; e.g.

/etc/fstab:
/data2/xray /srv/nfs/xray none defaults,bind 0

You can do that if you'd like. I doesn't make much difference here.

You can think of a filehandle as just a (device number, inode number) pair. (It's actually more complicated, but ignore that for now.)

So if the server's given a filehandle, it can easily determine the filehandle is for an object on /dev/sda2. It *cannot* easily determine whether that object is somewhere underneath /some/directory.

So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2.

Every export point should be the root of a filesystem.

--b.

Quoted 105 lines

Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.

Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file.  (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.  We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

#24 lore raw

FYI, if people really want to use some sort of subtree exports, for XFS (and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.

#25 lore raw
Quoted 5 lines · Christoph Hellwig

FYI, if people really want to use some sort of subtree exports, for XFS
(and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.

You'd basically need something along the lines of a NetApp qtree.

i.e. a persisted tag that can label all the files and directories in a subtree, and that is used to enforce a set of rules that are generally normally associated with filesystems. So no renames from objects inside the subtree to directories that lie outside it. No hard links that cross the subtree boundary.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#26 lore raw
Quoted 7 lines · Trond Myklebust
Quoted 5 lines · Christoph Hellwig

FYI, if people really want to use some sort of subtree exports, for XFS
(and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.

Quoted 7 lines

You'd basically need something along the lines of a NetApp qtree.

i.e. a persisted tag that can label all the files and directories in a subtree, and that is used to enforce a set of rules that are generally normally associated with filesystems. So no renames from objects inside the subtree to directories that lie outside it. No hard links that cross the subtree boundary.

That is the XFS project ID, which ext4 has also picked up a few years ago.

#27 lore raw
Quoted 22 lines · On Wed, 2021-01-13 at 14:40 +0000, [email protected] wrote:
Quoted 12 lines · Trond Myklebust
Quoted 8 lines · Christoph Hellwig

FYI, if people really want to use some sort of subtree exports, for
XFS
(and probably ext4) we encode the project id into the file handle and
use the hierarchical project ID inheritance that we already use for
project quotas.

You'd basically need something along the lines of a NetApp qtree.

Quoted 8 lines

i.e. a persisted tag that can label all the files and directories in a
subtree, and that is used to enforce a set of rules that are generally
normally associated with filesystems. So no renames from objects inside
the subtree to directories that lie outside it. No hard links that cross the subtree boundary.

Quoted 2 lines

That is the XFS project ID, which ext4 has also picked up a few years ago.

How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.

How do you set a project ID in XFS?

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#28 lore raw
Quoted 4 lines · Trond Myklebust

How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.

True, we would not even have to encode them.

Quoted 1 line

How do you set a project ID in XFS?

With the XFS_IOC_SETXFLAGS ioctl.

On the command line side people usually do it using the xfs_quota tool as part of setting up the tree quotas, but it can also be done separately using the chproj subcommand of xfs_io.

#29 lore raw
Quoted 6 lines · Trond Myklebust
Quoted 4 lines

How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.

Quoted 7 lines

True, we would not even have to encode them.

Quoted 1 line

How do you set a project ID in XFS?

With the XFS_IOC_SETXFLAGS ioctl.

On the command line side people usually do it using the xfs_quota tool as

part

Quoted 1 line

of setting up the tree quotas, but it can also be done separately using

the chproj

Quoted 1 line

subcommand of xfs_io.

Is this also queried via the ioctl? If this is a viable way of specifying sub-trees, nfs-ganesha could also use it, though making an ioctl call for each file system object would add some metadata performance hit.

Frank

#30 lore raw
In reply to [email protected]

See below.

Quoted 20 lines · On 1/12/21 12:03 PM, [email protected] wrote:
Quoted 9 lines · Patrick Goetz

I was under the impression that the best practice is to create something along the lines of

/srv/nfs

and then bind mount everything you plan to export into that folder; e.g.

/etc/fstab:
/data2/xray /srv/nfs/xray none defaults,bind 0

You can do that if you'd like. I doesn't make much difference here.

You can think of a filehandle as just a (device number, inode number) pair. (It's actually more complicated, but ignore that for now.)

So if the server's given a filehandle, it can easily determine the filehandle is for an object on /dev/sda2. It *cannot* easily determine whether that object is somewhere underneath /some/directory.

Quoted 8 lines

So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2.

Every export point should be the root of a filesystem.

--b.

I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in /etc/fstab:

/data2/xray /srv/nfs/xray none defaults,bind 0

it's my understanding that the kernel keeps track of the resulting /srv/nfs/xray filesystem in it's vfs somehow. Even when directly on the server I can't "break out" of /srv/nfs/xray to get to the other directories in /data. Then how on earth would an NFS client do this?

I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,

"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."

Yes, sure; but I'm not exporting /data2/xray; I'm exporting /srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?
It's not practical to making everything you export its own partition;
although I suppose one could do this with ZFS datasets.

Quoted 105 lines
Quoted 105 lines

Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.

Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file.  (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.  We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

#31 lore raw
Quoted 11 lines · Patrick Goetz

I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in /etc/fstab:

/data2/xray /srv/nfs/xray none defaults,bind 0

it's my understanding that the kernel keeps track of the resulting /srv/nfs/xray filesystem in it's vfs somehow. Even when directly on the server I can't "break out" of /srv/nfs/xray to get to the other directories in /data. Then how on earth would an NFS client do this?

As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path.

Also, note, mounting something over a directory doesn't hide what's under the mountpoint. And it's unwise to depend on directory permissions alone to hide contents of anything underneath that directory.

Quoted 8 lines

I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,

"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."

Quoted 5 lines

Yes, sure; but I'm not exporting /data2/xray; I'm exporting /srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?

Definitely do *not* depend on NFS to enforce different export options on different subdirectories of the same filesystem.

Quoted 2 lines

It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.

I'd be happy to hear about any use cases where that's not practical.

As Christophe pointed out, xfs/ext4 project ids are another option.

--b.

#32 lore raw
Quoted 7 lines · On 1/21/21 4:04 PM, [email protected] wrote:
Quoted 6 lines · Patrick Goetz

I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in /etc/fstab:

/data2/xray /srv/nfs/xray none defaults,bind 0

Quoted 8 lines
Quoted 5 lines

it's my understanding that the kernel keeps track of the resulting /srv/nfs/xray filesystem in it's vfs somehow. Even when directly on the server I can't "break out" of /srv/nfs/xray to get to the other directories in /data. Then how on earth would an NFS client do this?

As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path

Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with no_root_squash?

I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.

Quoted 5 lines

Also, note, mounting something over a directory doesn't hide what's under the mountpoint. And it's unwise to depend on directory permissions alone to hide contents of anything underneath that directory.

Well, I only ever bind mount over empty directories; but again, "doesn't hide what's under the mount point" from whom? I'm sure root can get to this somehow, but can someone with ordinary user access? even if the user doesn't have permissions to access the stuff that's been mounted over?

Quoted 15 lines
Quoted 9 lines

I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,

"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."

Quoted 5 lines

Yes, sure; but I'm not exporting /data2/xray; I'm exporting /srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?

Quoted 8 lines

Definitely do *not* depend on NFS to enforce different export options on different subdirectories of the same filesystem.

Quoted 2 lines

It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.

I'd be happy to hear about any use cases where that's not practical.

Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single device: /dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..

Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.

Quoted 4 lines

As Christophe pointed out, xfs/ext4 project ids are another option.

--b.

I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount? Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.

#33 lore raw
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:

As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path

Quoted 8 lines

Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with no_root_squash?

I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.

If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file. So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.

But it doesn't let them bypass the permissions on the file itself once they get there. If the permissions on the file don't allow read, the server still won't let them read it.

Quoted 21 lines
Quoted 4 lines
Quoted 2 lines

It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.

I'd be happy to hear about any use cases where that's not practical.

Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device: /dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..

Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.

Quoted 5 lines
Quoted 1 line

As Christophe pointed out, xfs/ext4 project ids are another option.

I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?

Right. The project ID is stored in the inode, so it's easy to look up from the filehandle. (Whereas figuring out what paths might lead to that inode is a little tricker.)

Quoted 3 lines

Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.

Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.

That said, I think Christoph was suggesting this is something we *could* support, not something that we now do. Though looking at it quickly, I think it shouldn't take much code at all. I'll put it on my list....

Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning. I haven't personally used either, but they should both work now.

--b.

#34 lore raw

Thanks for engaging; this has been informative.

Quoted 20 lines · On 1/21/21 7:30 PM, [email protected] wrote:
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:

As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path

Quoted 8 lines

Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with no_root_squash?

I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.

If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file. So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.

Quoted 4 lines

But it doesn't let them bypass the permissions on the file itself once they get there. If the permissions on the file don't allow read, the server still won't let them read it.

That's probably good enough. Security through obscurity isn't a good idea, so file/directory level permissions should be atomically correct and not rely on directory hierarchies, restricted direct access by users, or anything like this.

I didn't not know about the filehandle guessing thing and will keep that in mind for the next NFS server I deploy.

Quoted 48 lines
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines

It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.

I'd be happy to hear about any use cases where that's not practical.

Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device: /dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..

Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.

Quoted 5 lines
Quoted 1 line

As Christophe pointed out, xfs/ext4 project ids are another option.

I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?

Right. The project ID is stored in the inode, so it's easy to look up from the filehandle. (Whereas figuring out what paths might lead to that inode is a little tricker.)

Quoted 3 lines

Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.

Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.

That said, I think Christoph was suggesting this is something we *could* support, not something that we now do. Though looking at it quickly, I think it shouldn't take much code at all. I'll put it on my list....

Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning. I haven't personally used either, but they should both work now.

--b.

reviewer
#35 lore raw
Quoted 23 lines · Patrick Goetz

Thanks for engaging; this has been informative.

Quoted 20 lines · On 1/21/21 7:30 PM, [email protected] wrote:
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:

As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path

Quoted 8 lines

Except surely this doesn't buy you much if you don't have root access to the system?  Is this all only an issue when the filesystems are exported with no_root_squash?

I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.

If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file.  So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.

Quoted 8 lines
Quoted 4 lines

But it doesn't let them bypass the permissions on the file itself once they get there.  If the permissions on the file don't allow read, the server still won't let them read it.

That's probably good enough. Security through obscurity isn't a good idea, so file/directory level permissions should be atomically correct and not rely on directory hierarchies, restricted direct access by users, or anything like this.

I didn't not know about the filehandle guessing thing and will keep that in mind for the next NFS server I deploy.

There are some NFS clients which implement open-by-filehandle, and don't rely on lookup at all in normal operation. It is highly efficient, and provides a very low latency file access. I believe Yahoo was the first to implement it at scale in production, but it's a very straightforward cache operation.

Tom.

Quoted 49 lines
Quoted 48 lines
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines

It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.

I'd be happy to hear about any use cases where that's not practical.

Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device: /dev/sdg.  Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..

Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible.  I'm hoping zfs send/receive is going to save me here.

Quoted 5 lines
Quoted 1 line

As Christophe pointed out, xfs/ext4 project ids are another option.

I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?

Right.  The project ID is stored in the inode, so it's easy to look up from the filehandle.  (Whereas figuring out what paths might lead to that inode is a little tricker.)

Quoted 3 lines

Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.

Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.

That said, I think Christoph was suggesting this is something we *could* support, not something that we now do.  Though looking at it quickly, I think it shouldn't take much code at all.  I'll put it on my list....

Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning.  I haven't personally used either, but they should both work now.

--b.

A message that is not in the archive (sent off-list or not yet received): CAHxDmpTEBJ1jd_fr3GJ4k7KgzaBpe1LwKgyZn0AJ0D1ESK12fQ@mail.gmail.com

#37 lore raw
Quoted 4 lines · 吴异

Hello,

Well,maybe the best method is to prohibit  exporting subdirectiries,and I don't know how difficult it will be.

So, there is a discussion of all this in the 'exports' manpage both in the description of the 'no_subtree_check' option, and in the section on 'Subdirectory Exports'.
In particular, the latter section does describe the issue that you are raising here:

Subdirectory Exports
Normally you should only export only the root of a filesystem. The NFS server will also allow you to export a subdirectory of a filesystem, however, this has drawbacks:

First, it may be possible for a malicious user to access files on the filesystem outside of the exported subdirectory, by guessing filehan‐ dles for those other files. The only way to prevent this is by using the no_subtree_check option, which can cause other problems.

Second, export options may not be enforced in the way that you would expect. For example, the security_label option will not work on subdi‐ rectory exports, and if nested subdirectory exports change the secu‐ rity_label or sec= options, NFSv4 clients will normally see only the options on the parent export. Also, where security options differ, a malicious client may use filehandle-guessing attacks to access the files from one subdirectory using the options from another.

Why do we need to go further than this, when there are clearly also a load of scenarios where the clients are all trusted, and the security issue is moot?

Quoted 131 lines

Thanks,

Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异

Telling users how to configure the exported file system in the

most

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my opinion,when
the generated filehandle does not contain the inode information

of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the
parent directory,the nfsd_acceptable function can also

recursively

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings

some

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file.  (And note a single file

may

Quoted 4 lines
Quoted 4 lines

have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.

We

Quoted 10 lines
Quoted 10 lines

already reconnect directories all the way back up to the root.)

Quoted 8 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of

the

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of

the

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where

they're

Quoted 3 lines
Quoted 3 lines

being used.)

But given the interface we already have, that would be a

disruptive

Quoted 8 lines
Quoted 8 lines

and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 3 lines
Quoted 3 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can

do.

Quoted 3 lines
Quoted 3 lines

The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles

to

Quoted 5 lines
Quoted 3 lines

making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to

export a

Quoted 1 line

subtree of a filesystem' problem, and so it is plainly wrong to try

to

Quoted 5 lines

make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O

by

Quoted 1 line

invalidating the filehandle. In addition, that option causes

filehandle

Quoted 1 line

aliasing (i.e. multiple filehandles pointing to the same file)

which is

Quoted 5 lines

a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to

address

Quoted 6 lines

some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane

case.

Quoted 9 lines

That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

Signature · 6 lines
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022
​
www.hammer.space

A message that is not in the archive (sent off-list or not yet received): CAHxDmpTyrG74hOkzmDK834t+JiQduWHVWxCf_7nrDVa++EK2mA@mail.gmail.com

#39 lore raw
Quoted 12 lines · 吴异

Linux reference manual does describe the risk of exporting subdirectories of a file system,but I think there are some omissions.

/fs on /dev/sda1
export /fs/nfs

I can easily guess the handle of /fs,and then use it to remove the export point /nfs by a symlink,which is pointing to files in other file systems,e.g. "/" on /dev/sdb,it just like inserting a back door into the server.And then when nfsd service restart,I can mount to "/" ,this breaks file system isolation.

Quoted 4 lines

My question is whether Linux has made users fully aware of the security risks posed by subdirectories.In other words,how many people will be attacked with the nfsd when attack method is disclosed to the public?

The above is precisely what the manpage warns you about and tells you not to do as an administrator of a server.

BTW: you just 'disclosed the attack method to the public' since [email protected] is a public archived mailing list. However so far, you've said absolutely nothing that hasn't already been known and discussed for over 20 years.

Quoted 232 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 8 lines · 吴异
Quoted 4 lines

Hello,

Well,maybe the best method is to prohibit  exporting subdirectiries,and I don't know how difficult it will be.

So, there is a discussion of all this in the 'exports' manpage both

in

Quoted 1 line

the description of the 'no_subtree_check' option, and in the

section on

Quoted 2 lines

'Subdirectory Exports'.
In particular, the latter section does describe the issue that you

are

Quoted 4 lines

raising here:

   Subdirectory Exports
       Normally you should only export only the root of a

filesystem.  The NFS

Quoted 1 line

       server will also allow you to export a subdirectory  of  a

filesystem,

Quoted 3 lines

       however, this has drawbacks:

       First,  it  may be possible for a malicious user to access

files on the

Quoted 1 line

       filesystem outside of the exported subdirectory, by

guessing  filehan‐

Quoted 1 line

       dles  for  those other files.  The only way to prevent this

is by using

Quoted 3 lines

       the no_subtree_check option, which can cause other problems.

       Second, export options may not be enforced in the way  that

you  would

Quoted 1 line

       expect.  For example, the security_label option will not

work on subdi‐

Quoted 1 line

       rectory exports, and if nested subdirectory exports  change

the  secu‐

Quoted 1 line

       rity_label  or  sec=  options, NFSv4 clients will normally

see only the

Quoted 1 line

       options on the parent export.  Also, where security options

differ,  a

Quoted 1 line

       malicious  client  may  use  filehandle-guessing  attacks to

access the

Quoted 4 lines

       files from one subdirectory using the options from another.

Why do we need to go further than this, when there are clearly also

a

Quoted 1 line

load of scenarios where the clients are all trusted, and the

security

Quoted 10 lines

issue is moot?

Quoted 7 lines

Thanks,

Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异

Telling users how to configure the exported file system in

the

Quoted 4 lines
Quoted 4 lines

most

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

secure
way does
mitigate the problem to some extent, but this does not seem

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

address the
security risks posed by no_ subtree_ check in the code. In

my

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

opinion,when
the generated filehandle does not contain the inode

information

Quoted 7 lines
Quoted 7 lines

of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the
parent directory,the nfsd_acceptable function can also

recursively

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only

brings

Quoted 5 lines
Quoted 5 lines

some

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

troubles.

Filesystems don't necessarily provide us with an efficient way

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

find
parent directories from any given file.  (And note a single

file

Quoted 7 lines
Quoted 7 lines

may

Quoted 4 lines
Quoted 4 lines

have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.

We

Quoted 1 line
Quoted 1 line

already reconnect directories all the way back up to the

root.)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory in nfsd_acceptable (line 38 to 54

in

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

/fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

turned on by
default is that it will cause problems when reading and

writing

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security

of

Quoted 3 lines
Quoted 3 lines

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

system
depends on the user's complete correct configuration(the

system

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

does not
prohibit the export of a subdirectory).

Enabling subtree_check to add parent directoryinformation

only

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

brings
some troubles.

In short,I think it's open to question whether the security

of

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

system depends on the user's complete correct

configuration(the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that

prohibited

Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line

subdirectory exports (or at least made it more obvious where

they're

Quoted 3 lines
Quoted 3 lines

being used.)

But given the interface we already have, that would be a

disruptive

Quoted 4 lines
Quoted 4 lines

and
time-consuming change.

Another approach is to add more entropy to filehandles so

they're

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In the end none of these change the fact that a filehandle has

an

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

infinite lifetime, so once it's leaked, there's nothing you

can

Quoted 3 lines
Quoted 3 lines

do.

Quoted 2 lines
Quoted 2 lines

The
authors suggest NFSv4 volatile filehandles as a solution to

that

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

problem, but I don't think they've thought through the

obstacles

Quoted 8 lines
Quoted 8 lines

to

Quoted 5 lines
Quoted 3 lines

making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to

export a

Quoted 1 line

subtree of a filesystem' problem, and so it is plainly wrong to

try

Quoted 3 lines
Quoted 3 lines

to

Quoted 2 lines

make a default of those solutions, which break the one sane case

of

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only

because a

Quoted 1 line
Quoted 1 line
Quoted 1 line

trivial rename of a file breaks the client's ability to perform

I/O

Quoted 6 lines
Quoted 6 lines

by

Quoted 1 line

invalidating the filehandle. In addition, that option causes

filehandle

Quoted 1 line

aliasing (i.e. multiple filehandles pointing to the same file)

which is

Quoted 1 line

a major PITA for clients to try to manage for more or less the

same

Quoted 1 line
Quoted 1 line
Quoted 1 line

reason that it is a major PITA to try to manage these files

using

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

paths.

The discussion on volatile filehandles in RFC5661 does try to

address

Quoted 1 line

some of the above issues, but ends up concluding that you need

to

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

introduce POSIX-incompatible restrictions, such as trying to ban
renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the

sane

Quoted 20 lines
Quoted 10 lines

case.

Quoted 9 lines

That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

--
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022

www.hammer.space

Signature · 6 lines
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022
​
www.hammer.space
#40 lore raw
Quoted 4 lines · Trond Myklebust

BTW: you just 'disclosed the attack method to the public' since [email protected] is a public archived mailing list. However so far, you've said absolutely nothing that hasn't already been known and discussed for over 20 years.

I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.

It's not a huge surprise either, and doesn't change our basic recommendation (normally you should only export the roots of filesystems). Which is why I asked the reporter to move the discussion to the public list.

I think we could do better here.

--b.

Quoted 242 lines
Quoted 233 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 8 lines · 吴异
Quoted 4 lines

Hello,

Well,maybe the best method is to prohibit  exporting subdirectiries,and I don't know how difficult it will be.

So, there is a discussion of all this in the 'exports' manpage both

in

Quoted 1 line

the description of the 'no_subtree_check' option, and in the

section on

Quoted 2 lines

'Subdirectory Exports'.
In particular, the latter section does describe the issue that you

are

Quoted 4 lines

raising here:

   Subdirectory Exports
       Normally you should only export only the root of a

filesystem.  The NFS

Quoted 1 line

       server will also allow you to export a subdirectory  of  a

filesystem,

Quoted 3 lines

       however, this has drawbacks:

       First,  it  may be possible for a malicious user to access

files on the

Quoted 1 line

       filesystem outside of the exported subdirectory, by

guessing  filehan‐

Quoted 1 line

       dles  for  those other files.  The only way to prevent this

is by using

Quoted 3 lines

       the no_subtree_check option, which can cause other problems.

       Second, export options may not be enforced in the way  that

you  would

Quoted 1 line

       expect.  For example, the security_label option will not

work on subdi‐

Quoted 1 line

       rectory exports, and if nested subdirectory exports  change

the  secu‐

Quoted 1 line

       rity_label  or  sec=  options, NFSv4 clients will normally

see only the

Quoted 1 line

       options on the parent export.  Also, where security options

differ,  a

Quoted 1 line

       malicious  client  may  use  filehandle-guessing  attacks to

access the

Quoted 4 lines

       files from one subdirectory using the options from another.

Why do we need to go further than this, when there are clearly also

a

Quoted 1 line

load of scenarios where the clients are all trusted, and the

security

Quoted 10 lines

issue is moot?

Quoted 7 lines

Thanks,

Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异

Telling users how to configure the exported file system in

the

Quoted 4 lines
Quoted 4 lines

most

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

secure
way does
mitigate the problem to some extent, but this does not seem

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

address the
security risks posed by no_ subtree_ check in the code. In

my

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

opinion,when
the generated filehandle does not contain the inode

information

Quoted 7 lines
Quoted 7 lines

of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the
parent directory,the nfsd_acceptable function can also

recursively

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only

brings

Quoted 5 lines
Quoted 5 lines

some

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

troubles.

Filesystems don't necessarily provide us with an efficient way

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

find
parent directories from any given file.  (And note a single

file

Quoted 7 lines
Quoted 7 lines

may

Quoted 4 lines
Quoted 4 lines

have multiple parent directories.)

(I do wonder if we could do better in the directory case, though.

We

Quoted 1 line
Quoted 1 line

already reconnect directories all the way back up to the

root.)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory in nfsd_acceptable (line 38 to 54

in

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

/fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

turned on by
default is that it will cause problems when reading and

writing

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security

of

Quoted 3 lines
Quoted 3 lines

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

system
depends on the user's complete correct configuration(the

system

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

does not
prohibit the export of a subdirectory).

Enabling subtree_check to add parent directoryinformation

only

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

brings
some troubles.

In short,I think it's open to question whether the security

of

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

system depends on the user's complete correct

configuration(the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that

prohibited

Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line

subdirectory exports (or at least made it more obvious where

they're

Quoted 3 lines
Quoted 3 lines

being used.)

But given the interface we already have, that would be a

disruptive

Quoted 4 lines
Quoted 4 lines

and
time-consuming change.

Another approach is to add more entropy to filehandles so

they're

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

harder
to guess; see e.g.:

        https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In the end none of these change the fact that a filehandle has

an

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

infinite lifetime, so once it's leaked, there's nothing you

can

Quoted 3 lines
Quoted 3 lines

do.

Quoted 2 lines
Quoted 2 lines

The
authors suggest NFSv4 volatile filehandles as a solution to

that

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

problem, but I don't think they've thought through the

obstacles

Quoted 8 lines
Quoted 8 lines

to

Quoted 5 lines
Quoted 3 lines

making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to

export a

Quoted 1 line

subtree of a filesystem' problem, and so it is plainly wrong to

try

Quoted 3 lines
Quoted 3 lines

to

Quoted 2 lines

make a default of those solutions, which break the one sane case

of

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only

because a

Quoted 1 line
Quoted 1 line
Quoted 1 line

trivial rename of a file breaks the client's ability to perform

I/O

Quoted 6 lines
Quoted 6 lines

by

Quoted 1 line

invalidating the filehandle. In addition, that option causes

filehandle

Quoted 1 line

aliasing (i.e. multiple filehandles pointing to the same file)

which is

Quoted 1 line

a major PITA for clients to try to manage for more or less the

same

Quoted 1 line
Quoted 1 line
Quoted 1 line

reason that it is a major PITA to try to manage these files

using

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

paths.

The discussion on volatile filehandles in RFC5661 does try to

address

Quoted 1 line

some of the above issues, but ends up concluding that you need

to

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

introduce POSIX-incompatible restrictions, such as trying to ban
renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the

sane

Quoted 20 lines
Quoted 10 lines

case.

Quoted 9 lines

That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

--
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022

www.hammer.space

Signature · 6 lines
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022
​
www.hammer.space
#41 lore raw
Quoted 4 lines · On Thu, Jan 14, 2021 at 10:08 AM [email protected] <[email protected]>…

I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.

[ Other people removed from cc, this is just a question about nfsd cleanliness ]

I missed if Trond's suggestion to at least fix up ".." to have the same filehandle as "." for the top export directory was done.

Because honestly, the whole "guessing file handles is easy" argument doesn't seem to cover the case that the client just does something wrong _by_mistake_, and this ends up then exposing the server unnecessarily that way.

It's one thing if you have an actively malicious client that is controlled by an attacker and that then makes up its own file handles.

It's another thing if you have a (benign) client that can be fooled to access files on the server that it shouldn't have.

So I think that from a pure cleanliness standpoint, the server shouldn't give the client a file handle that the client mustn't actually ever use! It's just a recipe for "oops, I didn't mean to do something bad, but by mistake..."

Hmm?

Linus

#42 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 1 line · On Thu, Jan 14, 2021 at 10:08 AM [email protected] <[email protected]>…
Quoted 8 lines
Quoted 3 lines

I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.

[ Other people removed from cc, this is just a question about nfsd cleanliness ]

I missed if Trond's suggestion to at least fix up ".." to have the same filehandle as "." for the top export directory was done.

If I understand your question correctly... there is a commit in linux-next that simply doesn't return any filehandle for ".." in the root directory.

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=for-next&id=51b2ee7d006a736a9126e8111d1f24e4fd0afaa6

I intend to send you a PR after a few more days of soak time, unless you'd like me to send it now.

--
Chuck Lever

#43 lore raw
Quoted 4 lines · Chuck Lever

If I understand your question correctly... there is a commit in linux-next that simply doesn't return any filehandle for ".." in the root directory.

Great.

Quoted 2 lines

I intend to send you a PR after a few more days of soak time, unless you'd like me to send it now.

No, no hurry. I was more just checking that Trond's suggestion didn't get lost in the discussion about guessing file handles.

Linus

#44 lore raw
In reply to Trond Myklebust

hello,

I want to consult you on what is the original intention of designing subtree_check and whether it is to solve the 'I want to export a subtree of a filesystem' problem.

As far as I know, when opening subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry != exp->ex_path.dentry && !IS_ROOT(tdentry))' in nfsd_acceptable can work well to Intercept handles beyond the export point.

This seems to delete code as follows in nfsfh.c could solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;

Or replace by follow:
if (exp->ex_path.dentry == exp->vfs_mount->mnt_root) return 1;

When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like subtree_check to try to verify the file handle.

I may not understand your design ideas.

Yours sincerely,

Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

#45 lore raw
Quoted 20 lines · 吴异

I want to consult you on what is the original intention of designing subtree_check and whether it is to solve the 'I want to export a subtree of a filesystem' problem.

As far as I know, when opening subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry != exp->ex_path.dentry && !IS_ROOT(tdentry))' in nfsd_acceptable can work well to Intercept handles beyond the export point.

This seems to delete code as follows in nfsfh.c could solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;

Or replace by follow:
if (exp->ex_path.dentry == exp->vfs_mount->mnt_root) return 1;

Quoted 5 lines

When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like subtree_check to try to verify the file handle.

Sorry, I honestly don't understand the question.

If you have a specific proposal, perhaps you could send a patch.

--b.

Quoted 111 lines

I may not understand your design ideas.

Yours sincerely,

Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

#46 lore raw

My patch is below:
/fs/nfsd/nfsfh.c

nfsd_acceptable(void expv,struct dentry *dentry)
{
- if(exp->ex_flags & NFSEXP_NOSUBTREEXHECK)
- return 1;

+ if(is_root_export(exp))
+ return 1;

+ /* If not subdirectory export, accept anything*/

}

Quoted 145 lines · 在 2021年1月19日星期二,[email protected] <[email protected]> 写道:
Quoted 20 lines · 吴异

I want to consult you on what is the original intention of designing subtree_check and whether it is to solve the 'I want to export a subtree of a filesystem' problem.

As far as I know, when opening subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry != exp->ex_path.dentry && !IS_ROOT(tdentry))' in nfsd_acceptable can work well to Intercept handles beyond the export point.

This seems to delete code as follows in nfsfh.c could solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;

Or replace by follow:
if (exp->ex_path.dentry == exp->vfs_mount->mnt_root) return 1;

Quoted 5 lines

When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like subtree_check to try to verify the file handle.

Sorry, I honestly don't understand the question.

If you have a specific proposal, perhaps you could send a patch.

--b.

Quoted 111 lines

I may not understand your design ideas.

Yours sincerely,

Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

#47 lore raw

Review

Quoted 10 lines · 吴异

My patch is below:
/fs/nfsd/nfsfh.c

nfsd_acceptable(void expv,struct dentry *dentry)
{

(quoted diff) +2 -2
-     if(exp->ex_flags & NFSEXP_NOSUBTREEXHECK)
-     return 1;
 
+     if(is_root_export(exp))
+       return 1;

So you end up doing the SUBTREECHECK checks in cases where the filehandle doesn't actually have any parent information. That means we may be stuck with no way to connect the file back up the export point, so this check will fail even though the file may be under the export point. So users probably get spurious ESTALE errors.

To reproduce you probably need to do something like open a file on the client, reboot the server, then try to access the open file again. The dentry cache will no longer know how to look up the parents after the reboot.

--b.

Quoted 154 lines

+ /* If not subdirectory export, accept anything*/

}

Quoted 145 lines · 在 2021年1月19日星期二,[email protected] <[email protected]> 写道:
Quoted 20 lines · 吴异

I want to consult you on what is the original intention of designing subtree_check and whether it is to solve the 'I want to export a subtree of a filesystem' problem.

As far as I know, when opening subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry != exp->ex_path.dentry && !IS_ROOT(tdentry))' in nfsd_acceptable can work well to Intercept handles beyond the export point.

This seems to delete code as follows in nfsfh.c could solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;

Or replace by follow:
if (exp->ex_path.dentry == exp->vfs_mount->mnt_root) return 1;

Quoted 5 lines

When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like subtree_check to try to verify the file handle.

Sorry, I honestly don't understand the question.

If you have a specific proposal, perhaps you could send a patch.

--b.

Quoted 111 lines

I may not understand your design ideas.

Yours sincerely,

Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异

Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed by no_ subtree_ check in the code. In my

Quoted 8 lines

opinion,when
the generated filehandle does not contain the inode information of the
parent directory,the nfsd_acceptable function can also recursively determine whether the request file exceeds the export path dentry.Enabling
subtree_check to add parent directory information only brings some troubles.

Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)

(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)

Quoted 19 lines

I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory in nfsd_acceptable (line 38 to 54 in /fs/nfsd/nfsfh.c) .

As far as I understand it, the reason why subtree_check is not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time when nfsd_acceptable.

In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).

Quoted 8 lines

Enabling subtree_check to add parent directoryinformation only brings
some troubles.

In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).

I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)

But given the interface we already have, that would be a disruptive and
time-consuming change.

Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:

https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html

Quoted 8 lines

In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.

--b.

The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.

Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.

The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.

None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]