nfsd vurlerability submit
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): CAHxDmpTKJfnhGY9CVupyVYhNCTDVKBB6KRwh-E6u_XEPJq4WJQ@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): X/[email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAHxDmpSp1LHzKD5uqbfi+jcnb+nFaAZbc5++E0oOvLsYvyYDpw@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAHxDmpSjwrcr_fqLJa5=Zo=xmbt2Eo9dcy6TQuoU8+F3yVVNhw@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAHxDmpR8S7NR8OU2nWJmWBdFU9a7wDuDnxviQ2E9RDOeW9fExg@mail.gmail.com
Quoted 5 lines · 吴异
Yes, the safest way for users is that the first exported file system is on a different hard disk with "\", and then does not contain subdirectories, so I seem to have nothing to do. However, I did not find these tips in the configuration files related and man exports . It is best to recommend the best configuration method to users if it is updated in the future.
I agree that the documentation needs improvement.
I also think we should warn by default when a user tries to export something with no_subtree_check that's not at a mountpoint.
(Really, "at a mountpoint" should be "at the root of a filesystem", but I don't know how to check for that, at least not from userspace. I'm not sure whether the warning belongs in the kernel or elsewhere.)
Quoted 4 lines
Back to the present situation, have you considered removing the following piece of code in the
nfsd_acceptablefunction:
if (exp->ex_flags&NFSEXP_NOSUBTREECHECK)
return 1;
That would turn on subtree_check unconditionally. Currently we default to no_subtree_check, and for good reason: subtree_check causes unexpected behavior, like ESTALE on attempts to access open files after cross-directory rename.
Quoted 7 lines
In this way, all problems seem to be solved. Even if an attacker can get or guess some file handles, these handles will not actually work.Fix
compose_entry_fhandnfsd_lookup_dentryfunctions is actually not that urgent.One more question, do I need to copy all the previous emails to [email protected], this is a bit too much, right?
I don't think that's necessary. To summarize issue you've raised; if a client exports only a subdirectory of a filesystem:
- NFSv2/v3 clients can discover the filehandle of the parent of the export point using readdir_plus. I agree that we should fix this; I'll follow up with a patch. (It's already not too hard to guess filehandles, though.)
- Once a single filehandle outside the export directory is found, it's easy to access the rest of the filesystem. This issue has been known for decades. But I don't think it's well-enough understood by users.
- If only a subdirectory of a filesystem is exported, an attacker might be able to gain access to *other* filesystems as well, by replacing some component along the exported path by a symlink and waiting for the server to restart. (In the v2/v3 case, export path symlinks are followed on the server side.)
The main defense against all of this is to *only* export the root of a filesystem. If you're exporting "/export/home/", then you should create a separate filesystem and mount that at "/export/home/".
But this isn't as well understood as it should be, and we should do a better job of preventing users from choosing dangerous configurations. And then mitigate the danger where we can.
--b.
Quoted 196 lines
Best regards,
Quoted 191 lines · J. Bruce Fields <[email protected]> 于2021年1月11日周一 上午4:28写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异
I do try it,because of the default
no_subtreecheckin nfsd, I candelete
Quoted 1 line
Quoted 1 line
/nfs as long as I can get the handle of /fs. This takes advantage of
the
Quoted 9 lines
Quoted 9 lines
vulnerability I submitted before. I have added my poc and traffic information in the attachment.
My server filesystem configuration is as follows:
/dev/sda3on / type ext4 (rw,relatime,errors=remount-ro)
/dev/sda1on /fs type ext4 (rw,relatime)Quoted 9 lines
Quoted 3 lines
The configuration of
/etc/exportis/fs/nfs *(rw,sync,no_root_squash,insecure)
Again, you're exporting a subdirectory, which we discourage. The safe and recommended configuration would be to mount
/dev/sda1on /fs/nfs and export /fs/nfs.Quoted 1 line
As you can see, the file system where the shared directory is located
has
Quoted 1 line
Quoted 1 line
been separated from the “\” directory.What I did was to delete the
mount
Quoted 4 lines
Quoted 1 line
point directory
It should not be possible to delete /fs/nfs/ while you had something mounted there.
Whoops, I'm sorry, there's only a mountpoint at /fs/, so, yes, you can probably delete /fs/nfs and replace it by a symlink, got it.
Sure, that's interesting. But, again, we don't recommend exporting filesystem subdirectories.
Good to keep the public nfs list ([email protected]) cc'd on these things, by the way.
--b.
Quoted 5 lines
--b.
Quoted 2 lines
and restart the server-side nfsd service. At this time, the server-side shared directory became "/", and the configuration
information
Quoted 8 lines
Quoted 8 lines
did not change.
For the convenience I set
no_root_squashon the server, but in fact, as long as the server gives the user the permission to write in the /fs directory, the above operation can be completed.Quoted 8 lines
Quoted 8 lines
Best regards,
Quoted 2 lines · J. Bruce Fields <[email protected]> 于2021年1月9日周六 上午12:44写道:
Quoted 1 line · 吴异
when the shared directory /fs/nfs of the server is mounted on the
device
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
/dev/sda3, and the "/" directory is mounted on the device
/dev/sda1, thenQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the file system of the shared directory and the "/" file system
are In
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
isolation, an attacker can obtain the ability of any file in the
/dev/sda3Quoted 1 line
file system by using a file handle, but cannot access the "/" file
system
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
by using an NFS file handle, even if he already knows the "/"
handle is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
"0100010000000000", which will return the error code
nfserr_stale.In this case, the attacker deletes the mount point directory
/fs/nfs in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the file system under its control., and create a symbolic link
with the
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
same name pointing to the "/" directory.
Access to the filesystem mounted at /fs/nfs/ should not give you the ability to delete the directory /fs/nfs.
Have you actually tried this?
Please explain what access you assume the attacker starts with, what they're able to do it, why you think that isn't expected, and please test to make sure your idea works.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
--b.
Quoted 2 lines
After nfsd re-parses the
configuration file/etc/exports, the attacker can remount it toachieve
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
real file system escape.
It seems that nfsd should strengthen the restrictions on symbolic
link
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
resolution in configuration files.
Best regards,
Quoted 2 lines · J. Bruce Fields <[email protected]> 于2021年1月8日周五 下午11:20写道:
Quoted 1 line · Greg KH
On Tue, Jan 05, 2021 at 11:56:33AM -0500, J. Bruce Fields
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · 吴异
Quoted 2 lines
It is not safe to export only the root of the file system, Attacker can use lookup ".." to get the file handle to the
".."
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
directory,
Quoted 2 lines
Quoted 2 lines
You have claimed the ability to read and write any file on
the file
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
server. But if you put /fs/nfs/ on a separate filesystem
then
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
that is
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
no longer possible. Perhaps you can learn the value of the
filehandle
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
/fs/, but you can't use that filehandle to gain additional
access
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
unless
Quoted 3 lines
Quoted 3 lines
something on that filesystem is already exported.
So, there may be some information disclosure here, I agree,
but
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
it's
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
relatively mild, given that filehandles are not well-kept
secrets
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
anyway.
Ok, to clarify, you are saying that this really isn't an issue
at all
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
with the kernel code, so there's nothing to do here?
Just trying to figure out if there's anything to actually work
on
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
here
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
or not.
I think there's one possibly interesting issue, but it doesn't
require
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
special handling. Reporter has raised a few issues:
- The one that might be interesting: clients may be able
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
discover filehandles that they shouldn't be able to
using
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
readdir_plus. On review of the code, I agree it's oddthat
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the logic in
compose_entry_fh()is different from thatin
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
nfsd_lookup_dentry(). I'd also prefer to take thatdiscussion
Quoted 1 line
Quoted 1 line
to [email protected], as this seems at most a
mild
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
information disclosure.
- If only part of a filesystem is exported, an attacker
can
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
gain
Quoted 1 line
Quoted 1 line
access to files outside the exported directory by
guessing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
filehandles. This issue has been known for decades.
If we
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
want to discuss mitigation strategies, I believe the
correct
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
forum is again [email protected], not [email protected].
- An attacker might replace an exported mountpoint by a
symlink
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
pointing elsewhere. That seems to assume the attacker
already
Quoted 1 line
Quoted 1 line
has a lot of access, so I'm not clear what advantage
they
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
gain
Quoted 4 lines
Quoted 4 lines
from this.
--b.
PATCH 1/1 nfsd4: readdirplus shouldn't return parent of export
From: "J. Bruce Fields" <[email protected]>
If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.
The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.
Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.
However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.
--- fs/nfsd/nfs3xdr.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
fs/nfsd/nfs3xdr.c +2 -1
@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,if (isdotent (name, namlen)) {if (namlen ==2 ) { dchild =dget_parent (dparent); -/* filesystem root - cannot return filehandle for ".." */ +/* +* Don't return filehandle for ".." if we're at
OA+ * the filesystem or export root:
+ */
if (dchild == dparent)
goto out;
+ if (dparent == exp->ex_path.dentry)
+ goto out;
} else
dchild = dget(dparent);
} else
Signature · 1 line
2.29.2
Re: [PATCH] nfsd4: readdirplus shouldn't return parent of export
Quoted 44 lines · J. Bruce Fields
From: "J. Bruce Fields" <[email protected]>
If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.
The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.
Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.
However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.
Reported-by 吴异 <[email protected]>Signed-off-by J. Bruce Fields <[email protected]>---
fs/nfsd/nfs3xdr.c| 7 ++++++-1 file changed, 6 insertions(+), 1 deletion(-)fs/nfsd/nfs3xdr.c +2
-1@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,if (isdotent (name, namlen)) {if (namlen ==2 ) { dchild =dget_parent (dparent); -/* filesystem root - cannot return filehandle for ".." */ +/* +* Don't return filehandle for ".." if we're at OA+ * the filesystem or export root:
fs/nfsd/nfs3xdr.c +1
-0+*/ if (dchild == dparent) goto out;fs/nfsd/nfs3xdr.c +2
-0+if (dparent == exp->ex_path.dentry) + goto out;
Quoted 5 lines
} else dchild = dget(dparent); } elseSignature · 1 line
2.29.2
Thanks for the fix!
I've replaced the Reported-by: tag and pushed this to my cel-next topic branch, and intend to submit it with the next 5.11 -rc pull request. See:
https://git.linux-nfs.org/?p=cel/cel-2.6.git;a=shortlog;h=refs/heads/cel-next
Is there additional context that should be added? A Link:
tag that points to the discussion on security@ perhaps?
Note there was some damage in the patch body: there's a spurious "OA" in the hunk that had to be removed before the patch would apply.
--
Chuck Lever
Re: [PATCH] nfsd4: readdirplus shouldn't return parent of export
Quoted 52 lines · Chuck Lever
Quoted 44 lines · J. Bruce Fields
From: "J. Bruce Fields" <[email protected]>
If you export a subdirectory of a filesystem, a READDIRPLUS on the root of that export will return the filehandle of the parent with the ".." entry.
The filehandle is optional, so let's just not return the filehandle for ".." if we're at the root of an export.
Note that once the client learns one filehandle outside of the export, they can trivially access the rest of the export using further lookups.
However, it is also not very difficult to guess filehandles outside of the export. So exporting a subdirectory of a filesystem should considered equivalent to providing access to the entire filesystem. To avoid confusion, we recommend only exporting entire filesystems.
Reported-by 吴异 <[email protected]>Signed-off-by J. Bruce Fields <[email protected]>---
fs/nfsd/nfs3xdr.c| 7 ++++++-1 file changed, 6 insertions(+), 1 deletion(-)fs/nfsd/nfs3xdr.c +2
-1@@ -865,9 +865,14 @@ compose_entry_fh(struct nfsd3_readdirres *cd, struct svc_fh *fhp,if (isdotent (name, namlen)) {if (namlen ==2 ) { dchild =dget_parent (dparent); -/* filesystem root - cannot return filehandle for ".." */ +/* +* Don't return filehandle for ".." if we're at OA+ * the filesystem or export root:
fs/nfsd/nfs3xdr.c +1
-0+*/ if (dchild == dparent) goto out;fs/nfsd/nfs3xdr.c +2
-0+if (dparent == exp->ex_path.dentry) + goto out;Quoted 5 lines
} else dchild = dget(dparent); } elseSignature · 1 line
2.29.2Thanks for the fix!
Quoted 8 lines
I've replaced the Reported-by: tag and pushed this to my cel-next topic branch, and intend to submit it with the next 5.11 -rc pull request. See:
https://git.linux-nfs.org/?p=cel/cel-2.6.git;a=shortlog;h=refs/heads/cel-next
Is there additional context that should be added? A Link:
tag that points to the discussion on security@ perhaps?
I don't think so.
I guess it should get a stable cc: too.
Quoted 3 lines
Note there was some damage in the patch body: there's a spurious "OA" in the hunk that had to be removed before the patch would apply.
Whoops, apologies, I'm not sure how that happened....
--b.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAHxDmpR1zG25ADfK2jat4VKGbAOCg6YM_0WA+a_jQE82hbnMjA@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAHxDmpRfmVukMR_yF4coioiuzrsp72zBraHWZ8gaMydUuLwKFg@mail.gmail.com
Quoted 8 lines · 吴异
Telling users how to configure the exported file system in the most secure way does
mitigate the problem to some extent, but this does not seem to address the security risks posed byno_subtree_check in the code. In my opinion,when the generated filehandle does not contain the inode information of the parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.
Filesystems don't necessarily provide us with an efficient way to find parent directories from any given file. (And note a single file may have multiple parent directories.)
(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 13 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by default is that it will cause problems when reading and writing files, rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system does not prohibit the export of a subdirectory).
Quoted 6 lines
Enabling
subtree_checkto add parent directoryinformation only brings some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system does not prohibit the export of a subdirectory).
I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and time-consuming change.
Another approach is to add more entropy to filehandles so they're harder to guess; see e.g.:
https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.
--b.
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out subtree_check not only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.
The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
I was under the impression that the best practice is to create something along the lines of
/srv/nfs
and then bind mount everything you plan to export into that folder; e.g.
/etc/fstab:
/data2/xray /srv/nfs/xray none defaults,bind 0
Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.
Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
Quoted 9 lines · Patrick Goetz
I was under the impression that the best practice is to create something along the lines of
/srv/nfsand then bind mount everything you plan to export into that folder; e.g.
/etc/fstab:
/data2/xray/srv/nfs/xraynone defaults,bind 0
You can do that if you'd like. I doesn't make much difference here.
You can think of a filehandle as just a (device number, inode number) pair. (It's actually more complicated, but ignore that for now.)
So if the server's given a filehandle, it can easily determine the filehandle is for an object on /dev/sda2. It *cannot* easily determine whether that object is somewhere underneath /some/directory.
So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2.
Every export point should be the root of a filesystem.
--b.
Quoted 105 lines
Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.
Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
FYI, if people really want to use some sort of subtree exports, for XFS (and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.
Quoted 5 lines · Christoph Hellwig
FYI, if people really want to use some sort of subtree exports, for XFS
(and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.
You'd basically need something along the lines of a NetApp qtree.
i.e. a persisted tag that can label all the files and directories in a subtree, and that is used to enforce a set of rules that are generally normally associated with filesystems. So no renames from objects inside the subtree to directories that lie outside it. No hard links that cross the subtree boundary.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 7 lines · Trond Myklebust
Quoted 5 lines · Christoph Hellwig
FYI, if people really want to use some sort of subtree exports, for XFS
(and probably ext4) we encode the project id into the file handle and use the hierarchical project ID inheritance that we already use for project quotas.
Quoted 7 lines
You'd basically need something along the lines of a NetApp qtree.
i.e. a persisted tag that can label all the files and directories in a subtree, and that is used to enforce a set of rules that are generally normally associated with filesystems. So no renames from objects inside the subtree to directories that lie outside it. No hard links that cross the subtree boundary.
That is the XFS project ID, which ext4 has also picked up a few years ago.
Quoted 22 lines · On Wed, 2021-01-13 at 14:40 +0000, [email protected] wrote:
Quoted 12 lines · Trond Myklebust
Quoted 8 lines · Christoph Hellwig
FYI, if people really want to use some sort of subtree exports, for
XFS
(and probably ext4) we encode the project id into the file handle and
use the hierarchical project ID inheritance that we already use for
project quotas.You'd basically need something along the lines of a NetApp qtree.
Quoted 8 lines
i.e. a persisted tag that can label all the files and directories in a
subtree, and that is used to enforce a set of rules that are generally
normally associated with filesystems. So no renames from objects inside
the subtree to directories that lie outside it. No hard links that cross the subtree boundary.
Quoted 2 lines
That is the XFS project ID, which ext4 has also picked up a few years ago.
How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.
How do you set a project ID in XFS?
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 4 lines · Trond Myklebust
How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.
True, we would not even have to encode them.
Quoted 1 line
How do you set a project ID in XFS?
With the XFS_IOC_SETXFLAGS ioctl.
On the command line side people usually do it using the xfs_quota tool as part of setting up the tree quotas, but it can also be done separately using the chproj subcommand of xfs_io.
Quoted 6 lines · Trond Myklebust
Quoted 4 lines
How would that work then? Would you just look at the project ID of the directory identified by the filehandle as the export point, and then match to the project ID on the target inode? That sounds like it doesn't even need to encode anything special in the filehandle.
Quoted 7 lines
True, we would not even have to encode them.
Quoted 1 line
How do you set a project ID in XFS?
With the
XFS_IOC_SETXFLAGSioctl.On the command line side people usually do it using the
xfs_quotatool as
part
Quoted 1 line
of setting up the tree quotas, but it can also be done separately using
the chproj
Quoted 1 line
subcommand of
xfs_io.
Is this also queried via the ioctl? If this is a viable way of specifying sub-trees, nfs-ganesha could also use it, though making an ioctl call for each file system object would add some metadata performance hit.
Frank
See below.
Quoted 20 lines · On 1/12/21 12:03 PM, [email protected] wrote:
Quoted 9 lines · Patrick Goetz
I was under the impression that the best practice is to create something along the lines of
/srv/nfsand then bind mount everything you plan to export into that folder; e.g.
/etc/fstab:
/data2/xray/srv/nfs/xraynone defaults,bind 0You can do that if you'd like. I doesn't make much difference here.
You can think of a filehandle as just a (device number, inode number) pair. (It's actually more complicated, but ignore that for now.)
So if the server's given a filehandle, it can easily determine the filehandle is for an object on
/dev/sda2. It *cannot* easily determine whether that object is somewhere underneath /some/directory.
Quoted 8 lines
So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2.
Every export point should be the root of a filesystem.
--b.
I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in /etc/fstab:
/data2/xray /srv/nfs/xray none defaults,bind 0
it's my understanding that the kernel keeps track of the resulting /srv/nfs/xray filesystem in it's vfs somehow. Even when directly on the server I can't "break out" of /srv/nfs/xray to get to the other directories in /data. Then how on earth would an NFS client do this?
I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,
"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."
Yes, sure; but I'm not exporting /data2/xray; I'm exporting /srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?
It's not practical to making everything you export its own partition;
although I suppose one could do this with ZFS datasets.
Quoted 105 lines
Quoted 105 lines
Presumably this becomes a non-issue under these circumstances? Not sure it's a good idea to attempt to accommodate every wacky use case someone attempts to implement.
Quoted 98 lines · Trond Myklebust
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
Quoted 11 lines · Patrick Goetz
I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in
/etc/fstab:/data2/xray
/srv/nfs/xraynone defaults,bind 0it's my understanding that the kernel keeps track of the resulting
/srv/nfs/xrayfilesystem in it's vfs somehow. Even when directly on the server I can't "break out" of/srv/nfs/xrayto get to the other directories in /data. Then how on earth would an NFS client do this?
As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path.
Also, note, mounting something over a directory doesn't hide what's under the mountpoint. And it's unwise to depend on directory permissions alone to hide contents of anything underneath that directory.
Quoted 8 lines
I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,
"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."
Quoted 5 lines
Yes, sure; but I'm not exporting /data2/xray; I'm exporting
/srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?
Definitely do *not* depend on NFS to enforce different export options on different subdirectories of the same filesystem.
Quoted 2 lines
It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.
I'd be happy to hear about any use cases where that's not practical.
As Christophe pointed out, xfs/ext4 project ids are another option.
--b.
Quoted 7 lines · On 1/21/21 4:04 PM, [email protected] wrote:
Quoted 6 lines · Patrick Goetz
I didn't respond to this message immediately, but it's been bothering me ever since. When I do a bind mount like this in
/etc/fstab:/data2/xray
/srv/nfs/xraynone defaults,bind 0
Quoted 8 lines
Quoted 5 lines
it's my understanding that the kernel keeps track of the resulting
/srv/nfs/xrayfilesystem in it's vfs somehow. Even when directly on the server I can't "break out" of/srv/nfs/xrayto get to the other directories in /data. Then how on earth would an NFS client do this?As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path
Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with no_root_squash?
I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.
Quoted 5 lines
Also, note, mounting something over a directory doesn't hide what's under the mountpoint. And it's unwise to depend on directory permissions alone to hide contents of anything underneath that directory.
Well, I only ever bind mount over empty directories; but again, "doesn't hide what's under the mount point" from whom? I'm sure root can get to this somehow, but can someone with ordinary user access? even if the user doesn't have permissions to access the stuff that's been mounted over?
Quoted 15 lines
Quoted 9 lines
I thought the whole point of doing a bind mount like this is to solve the problem of exporting leaves of a directory hierarchy. In particular,
"So in your example, if /data2/xray is on the same filesystem as /data2, then the server will happily allow operations on filehandles anywhere in /data2."
Quoted 5 lines
Yes, sure; but I'm not exporting /data2/xray; I'm exporting
/srv/nfs/xray, a bind mount to the preceding. Am I missing something, or is NFS too insecure to use in any context requiring differentiated security settings on different folders in the same directory structure?
Quoted 8 lines
Definitely do *not* depend on NFS to enforce different export options on different subdirectories of the same filesystem.
Quoted 2 lines
It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.
I'd be happy to hear about any use cases where that's not practical.
Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single device: /dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..
Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.
Quoted 4 lines
As Christophe pointed out, xfs/ext4 project ids are another option.
--b.
I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount? Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:
As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path
Quoted 8 lines
Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with
no_root_squash?I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.
If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file. So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.
But it doesn't let them bypass the permissions on the file itself once they get there. If the permissions on the file don't allow read, the server still won't let them read it.
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines
It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.
I'd be happy to hear about any use cases where that's not practical.
Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device:/dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.
Quoted 5 lines
Quoted 1 line
As Christophe pointed out, xfs/ext4 project ids are another option.
I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?
Right. The project ID is stored in the inode, so it's easy to look up from the filehandle. (Whereas figuring out what paths might lead to that inode is a little tricker.)
Quoted 3 lines
Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.
Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.
That said, I think Christoph was suggesting this is something we *could* support, not something that we now do. Though looking at it quickly, I think it shouldn't take much code at all. I'll put it on my list....
Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning. I haven't personally used either, but they should both work now.
--b.
Thanks for engaging; this has been informative.
Quoted 20 lines · On 1/21/21 7:30 PM, [email protected] wrote:
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:
As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path
Quoted 8 lines
Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with
no_root_squash?I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.
If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file. So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.
Quoted 4 lines
But it doesn't let them bypass the permissions on the file itself once they get there. If the permissions on the file don't allow read, the server still won't let them read it.
That's probably good enough. Security through obscurity isn't a good idea, so file/directory level permissions should be atomically correct and not rely on directory hierarchies, restricted direct access by users, or anything like this.
I didn't not know about the filehandle guessing thing and will keep that in mind for the next NFS server I deploy.
Quoted 48 lines
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines
It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.
I'd be happy to hear about any use cases where that's not practical.
Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device:/dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.
Quoted 5 lines
Quoted 1 line
As Christophe pointed out, xfs/ext4 project ids are another option.
I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?
Right. The project ID is stored in the inode, so it's easy to look up from the filehandle. (Whereas figuring out what paths might lead to that inode is a little tricker.)
Quoted 3 lines
Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.
Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.
That said, I think Christoph was suggesting this is something we *could* support, not something that we now do. Though looking at it quickly, I think it shouldn't take much code at all. I'll put it on my list....
Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning. I haven't personally used either, but they should both work now.
--b.
Tom Talpey
reviewerQuoted 23 lines · Patrick Goetz
Thanks for engaging; this has been informative.
Quoted 20 lines · On 1/21/21 7:30 PM, [email protected] wrote:
Quoted 4 lines · Patrick Goetz
Quoted 2 lines · On 1/21/21 4:04 PM, [email protected] wrote:
As I said, NFS allows you to look up objects by filehandle (so, basically by inode number), not just by path
Quoted 8 lines
Except surely this doesn't buy you much if you don't have root access to the system? Is this all only an issue when the filesystems are exported with
no_root_squash?I feel like I must be missing something, but it seems to me that if I'm not root, I'm not going to be able to access inodes I don't have permissions to access even when directly connected to the exporting server.
If an attacker has access to the network (so they can send their own hand-crafted NFS requests), then filehandle guessing allows them to bypass the normal process of looking up a file. So if you were depending on lookup permissions along that path, or on hiding that path somehow, you're out of luck.
Quoted 8 lines
Quoted 4 lines
But it doesn't let them bypass the permissions on the file itself once they get there. If the permissions on the file don't allow read, the server still won't let them read it.
That's probably good enough. Security through obscurity isn't a good idea, so file/directory level permissions should be atomically correct and not rely on directory hierarchies, restricted direct access by users, or anything like this.
I didn't not know about the filehandle guessing thing and will keep that in mind for the next NFS server I deploy.
There are some NFS clients which implement open-by-filehandle, and don't rely on lookup at all in normal operation. It is highly efficient, and provides a very low latency file access. I believe Yahoo was the first to implement it at scale in production, but it's a very straightforward cache operation.
Tom.
Quoted 49 lines
Quoted 48 lines
Quoted 21 lines
Quoted 4 lines
Quoted 2 lines
It's not practical to making everything you export its own partition; although I suppose one could do this with ZFS datasets.
I'd be happy to hear about any use cases where that's not practical.
Sure. The xray example is taken from one of my research groups which collects thousands of very large electron microscopy images, along with some xray data. I will certainly design this differently in the next iteration (most likely using ZFS), but our current server has a 519T attached storage device which presents itself as a single
device: /dev/sdg. Different groups need access to different classes of data, which I export separately and with are presented on the workstations as /xray, /EM, etc..Yes, I could partition the storage device, but then I run into the usual issues where one partition runs out of space while others are barely utilized. This is one good reason to switch to ZFS datasets. The other is that -- with 450T+ of ever changing data, currently rsync backups are almost impossible. I'm hoping zfs send/receive is going to save me here.
Quoted 5 lines
Quoted 1 line
As Christophe pointed out, xfs/ext4 project ids are another option.
I must have missed this one, but it just leaves me more confused. Project ID's are filesystem metadata, yet this affords better boundary enforcement than a bind mount?
Right. The project ID is stored in the inode, so it's easy to look up from the filehandle. (Whereas figuring out what paths might lead to that inode is a little tricker.)
Quoted 3 lines
Also, the only use case for Project ID's I was able to find are project quotas, so am not even sure how this would be implemented, and used by NFS.
Project ID's were implemented for quotas, but they also have the characteristics to work well as NFS export boundaries.
That said, I think Christoph was suggesting this is something we *could* support, not something that we now do. Though looking at it quickly, I think it shouldn't take much code at all. I'll put it on my list....
Other options for doing this kind of thing might be btrfs subvolumes or lvm thin provisioning. I haven't personally used either, but they should both work now.
--b.
A message that is not in the archive (sent off-list or not yet received): CAHxDmpTEBJ1jd_fr3GJ4k7KgzaBpe1LwKgyZn0AJ0D1ESK12fQ@mail.gmail.com
Quoted 4 lines · 吴异
Hello,
Well,maybe the best method is to prohibit exporting subdirectiries,and I don't know how difficult it will be.
So, there is a discussion of all this in the 'exports' manpage both in the description of the 'no_subtree_check' option, and in the section on 'Subdirectory Exports'.
In particular, the latter section does describe the issue that you are raising here:
Subdirectory Exports
Normally you should only export only the root of a filesystem. The NFS server will also allow you to export a subdirectory of a filesystem, however, this has drawbacks:
First, it may be possible for a malicious user to access files on the filesystem outside of the exported subdirectory, by guessing filehan‐ dles for those other files. The only way to prevent this is by using the no_subtree_check option, which can cause other problems.
Second, export options may not be enforced in the way that you would expect. For example, the security_label option will not work on subdi‐ rectory exports, and if nested subdirectory exports change the secu‐ rity_label or sec= options, NFSv4 clients will normally see only the options on the parent export. Also, where security options differ, a malicious client may use filehandle-guessing attacks to access the files from one subdirectory using the options from another.
Why do we need to go further than this, when there are clearly also a load of scenarios where the clients are all trusted, and the security issue is moot?
Quoted 131 lines
Thanks,
Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异
Telling users how to configure the exported file system in the
most
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In my opinion,when
the generated filehandle does not contain the inode informationof
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
parent directory,thenfsd_acceptablefunction can alsorecursively
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
determine whether the request file exceeds the export path dentry.Enabling
subtree_checkto add parent directory information only bringssome
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
troubles.
Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single filemay
Quoted 4 lines
Quoted 4 lines
have multiple parent directories.)
(I do wonder if we could do better in the directory case, though.
We
Quoted 10 lines
Quoted 10 lines
already reconnect directories all the way back up to the root.)
Quoted 8 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of
the
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of
the
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where
they're
Quoted 3 lines
Quoted 3 lines
being used.)
But given the interface we already have, that would be a
disruptive
Quoted 8 lines
Quoted 8 lines
and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 3 lines
Quoted 3 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can
do.
Quoted 3 lines
Quoted 3 lines
The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstaclesto
Quoted 5 lines
Quoted 3 lines
making volatile filehandles work.
--b.
The point is that there is no good solution to the 'I want to
export a
Quoted 1 line
subtree of a filesystem' problem, and so it is plainly wrong to try
to
Quoted 5 lines
make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/Oby
Quoted 1 line
invalidating the filehandle. In addition, that option causes
filehandle
Quoted 1 line
aliasing (i.e. multiple filehandles pointing to the same file)
which is
Quoted 5 lines
a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.
The discussion on volatile filehandles in RFC5661 does try to
address
Quoted 6 lines
some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane
case.
Quoted 9 lines
That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Signature · 6 lines
Trond Myklebust CTO, Hammerspace Inc 4984 El Camino Real, Suite 208 Los Altos, CA 94022 www.hammer.space
A message that is not in the archive (sent off-list or not yet received): CAHxDmpTyrG74hOkzmDK834t+JiQduWHVWxCf_7nrDVa++EK2mA@mail.gmail.com
Quoted 12 lines · 吴异
Linux reference manual does describe the risk of exporting subdirectories of a file system,but I think there are some omissions.
/fs on
/dev/sda1
export /fs/nfsI can easily guess the handle of /fs,and then use it to remove the export point /nfs by a symlink,which is pointing to files in other file systems,e.g. "/" on
/dev/sdb,itjust like inserting a back door into the server.And then when nfsd service restart,I can mount to "/" ,this breaks file system isolation.
Quoted 4 lines
My question is whether Linux has made users fully aware of the security risks posed by subdirectories.In other words,how many people will be attacked with the nfsd when attack method is disclosed to the public?
The above is precisely what the manpage warns you about and tells you not to do as an administrator of a server.
BTW: you just 'disclosed the attack method to the public' since [email protected] is a public archived mailing list. However so far, you've said absolutely nothing that hasn't already been known and discussed for over 20 years.
Quoted 232 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 8 lines · 吴异
Quoted 4 lines
Hello,
Well,maybe the best method is to prohibit exporting subdirectiries,and I don't know how difficult it will be.
So, there is a discussion of all this in the 'exports' manpage both
in
Quoted 1 line
the description of the '
no_subtree_check' option, and in thesection on
Quoted 2 lines
'Subdirectory Exports'.
In particular, the latter section does describe the issue that youare
Quoted 4 lines
raising here:
Subdirectory Exports
Normally you should only export only the root of afilesystem. The NFS
Quoted 1 line
server will also allow you to export a subdirectory of a
filesystem,
Quoted 3 lines
however, this has drawbacks:
First, it may be possible for a malicious user to access
files on the
Quoted 1 line
filesystem outside of the exported subdirectory, by
guessing filehan‐
Quoted 1 line
dles for those other files. The only way to prevent this
is by using
Quoted 3 lines
the
no_subtree_checkoption, which can cause other problems.Second, export options may not be enforced in the way that
you would
Quoted 1 line
expect. For example, the
security_labeloption will notwork on subdi‐
Quoted 1 line
rectory exports, and if nested subdirectory exports change
the secu‐
Quoted 1 line
rity_label or sec= options, NFSv4 clients will normally
see only the
Quoted 1 line
options on the parent export. Also, where security options
differ, a
Quoted 1 line
malicious client may use filehandle-guessing attacks to
access the
Quoted 4 lines
files from one subdirectory using the options from another.
Why do we need to go further than this, when there are clearly also
a
Quoted 1 line
load of scenarios where the clients are all trusted, and the
security
Quoted 10 lines
issue is moot?
Quoted 7 lines
Thanks,
Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异
Telling users how to configure the exported file system in
the
Quoted 4 lines
Quoted 4 lines
most
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
secure
way does
mitigate the problem to some extent, but this does not seemto
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
address the
security risks posed byno_subtree_check in the code. Inmy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
opinion,when
the generated filehandle does not contain the inodeinformation
Quoted 7 lines
Quoted 7 lines
of
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
parent directory,thenfsd_acceptablefunction can alsorecursively
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
determine whether the request file exceeds the export path dentry.Enabling
subtree_checkto add parent directory information onlybrings
Quoted 5 lines
Quoted 5 lines
some
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
troubles.
Filesystems don't necessarily provide us with an efficient way
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
find
parent directories from any given file. (And note a singlefile
Quoted 7 lines
Quoted 7 lines
may
Quoted 4 lines
Quoted 4 lines
have multiple parent directories.)
(I do wonder if we could do better in the directory case, though.
We
Quoted 1 line
Quoted 1 line
already reconnect directories all the way back up to the
root.)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory innfsd_acceptable(line 38 to 54in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
/fs/nfsd/nfsfh.c) .
As far as I understand it, the reason why
subtree_checkisnot
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
turned on by
default is that it will cause problems when reading andwriting
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security
of
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
system
depends on the user's complete correct configuration(thesystem
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
does not
prohibit the export of a subdirectory).Enabling
subtree_checkto add parent directoryinformationonly
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
brings
some troubles.In short,I think it's open to question whether the security
of
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
system depends on the user's complete correct
configuration(the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that
prohibited
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
subdirectory exports (or at least made it more obvious where
they're
Quoted 3 lines
Quoted 3 lines
being used.)
But given the interface we already have, that would be a
disruptive
Quoted 4 lines
Quoted 4 lines
and
time-consuming change.Another approach is to add more entropy to filehandles so
they're
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In the end none of these change the fact that a filehandle has
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
infinite lifetime, so once it's leaked, there's nothing you
can
Quoted 3 lines
Quoted 3 lines
do.
Quoted 2 lines
Quoted 2 lines
The
authors suggest NFSv4 volatile filehandles as a solution tothat
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
problem, but I don't think they've thought through the
obstacles
Quoted 8 lines
Quoted 8 lines
to
Quoted 5 lines
Quoted 3 lines
making volatile filehandles work.
--b.
The point is that there is no good solution to the 'I want to
export a
Quoted 1 line
subtree of a filesystem' problem, and so it is plainly wrong to
try
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
make a default of those solutions, which break the one sane case
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot onlybecause a
Quoted 1 line
Quoted 1 line
Quoted 1 line
trivial rename of a file breaks the client's ability to perform
I/O
Quoted 6 lines
Quoted 6 lines
by
Quoted 1 line
invalidating the filehandle. In addition, that option causes
filehandle
Quoted 1 line
aliasing (i.e. multiple filehandles pointing to the same file)
which is
Quoted 1 line
a major PITA for clients to try to manage for more or less the
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
reason that it is a major PITA to try to manage these files
using
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
paths.
The discussion on volatile filehandles in RFC5661 does try to
address
Quoted 1 line
some of the above issues, but ends up concluding that you need
to
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
introduce POSIX-incompatible restrictions, such as trying to ban
renames and deletions of open files in order to make it work.None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the
sane
Quoted 20 lines
Quoted 10 lines
case.
Quoted 9 lines
That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]--
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022www.hammer.space
Signature · 6 lines
Trond Myklebust CTO, Hammerspace Inc 4984 El Camino Real, Suite 208 Los Altos, CA 94022 www.hammer.space
Quoted 4 lines · Trond Myklebust
BTW: you just 'disclosed the attack method to the public' since [email protected] is a public archived mailing list. However so far, you've said absolutely nothing that hasn't already been known and discussed for over 20 years.
I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.
It's not a huge surprise either, and doesn't change our basic recommendation (normally you should only export the roots of filesystems). Which is why I asked the reporter to move the discussion to the public list.
I think we could do better here.
--b.
Quoted 242 lines
Quoted 233 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 8 lines · 吴异
Quoted 4 lines
Hello,
Well,maybe the best method is to prohibit exporting subdirectiries,and I don't know how difficult it will be.
So, there is a discussion of all this in the 'exports' manpage both
in
Quoted 1 line
the description of the '
no_subtree_check' option, and in thesection on
Quoted 2 lines
'Subdirectory Exports'.
In particular, the latter section does describe the issue that youare
Quoted 4 lines
raising here:
Subdirectory Exports
Normally you should only export only the root of afilesystem. The NFS
Quoted 1 line
server will also allow you to export a subdirectory of a
filesystem,
Quoted 3 lines
however, this has drawbacks:
First, it may be possible for a malicious user to access
files on the
Quoted 1 line
filesystem outside of the exported subdirectory, by
guessing filehan‐
Quoted 1 line
dles for those other files. The only way to prevent this
is by using
Quoted 3 lines
the
no_subtree_checkoption, which can cause other problems.Second, export options may not be enforced in the way that
you would
Quoted 1 line
expect. For example, the
security_labeloption will notwork on subdi‐
Quoted 1 line
rectory exports, and if nested subdirectory exports change
the secu‐
Quoted 1 line
rity_label or sec= options, NFSv4 clients will normally
see only the
Quoted 1 line
options on the parent export. Also, where security options
differ, a
Quoted 1 line
malicious client may use filehandle-guessing attacks to
access the
Quoted 4 lines
files from one subdirectory using the options from another.
Why do we need to go further than this, when there are clearly also
a
Quoted 1 line
load of scenarios where the clients are all trusted, and the
security
Quoted 10 lines
issue is moot?
Quoted 7 lines
Thanks,
Quoted 3 lines · 在 2021年1月13日星期三,Trond Myklebust <[email protected]> 写道:
Quoted 2 lines · J. Bruce Fields
Quoted 1 line · 吴异
Telling users how to configure the exported file system in
the
Quoted 4 lines
Quoted 4 lines
most
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
secure
way does
mitigate the problem to some extent, but this does not seemto
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
address the
security risks posed byno_subtree_check in the code. Inmy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
opinion,when
the generated filehandle does not contain the inodeinformation
Quoted 7 lines
Quoted 7 lines
of
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
parent directory,thenfsd_acceptablefunction can alsorecursively
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
determine whether the request file exceeds the export path dentry.Enabling
subtree_checkto add parent directory information onlybrings
Quoted 5 lines
Quoted 5 lines
some
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
troubles.
Filesystems don't necessarily provide us with an efficient way
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
find
parent directories from any given file. (And note a singlefile
Quoted 7 lines
Quoted 7 lines
may
Quoted 4 lines
Quoted 4 lines
have multiple parent directories.)
(I do wonder if we could do better in the directory case, though.
We
Quoted 1 line
Quoted 1 line
already reconnect directories all the way back up to the
root.)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry
exceeds
the export point directory innfsd_acceptable(line 38 to 54in
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
/fs/nfsd/nfsfh.c) .
As far as I understand it, the reason why
subtree_checkisnot
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
turned on by
default is that it will cause problems when reading andwriting
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security
of
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
system
depends on the user's complete correct configuration(thesystem
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
does not
prohibit the export of a subdirectory).Enabling
subtree_checkto add parent directoryinformationonly
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
brings
some troubles.In short,I think it's open to question whether the security
of
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
system depends on the user's complete correct
configuration(the
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that
prohibited
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
subdirectory exports (or at least made it more obvious where
they're
Quoted 3 lines
Quoted 3 lines
being used.)
But given the interface we already have, that would be a
disruptive
Quoted 4 lines
Quoted 4 lines
and
time-consuming change.Another approach is to add more entropy to filehandles so
they're
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In the end none of these change the fact that a filehandle has
an
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
infinite lifetime, so once it's leaked, there's nothing you
can
Quoted 3 lines
Quoted 3 lines
do.
Quoted 2 lines
Quoted 2 lines
The
authors suggest NFSv4 volatile filehandles as a solution tothat
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
problem, but I don't think they've thought through the
obstacles
Quoted 8 lines
Quoted 8 lines
to
Quoted 5 lines
Quoted 3 lines
making volatile filehandles work.
--b.
The point is that there is no good solution to the 'I want to
export a
Quoted 1 line
subtree of a filesystem' problem, and so it is plainly wrong to
try
Quoted 3 lines
Quoted 3 lines
to
Quoted 2 lines
make a default of those solutions, which break the one sane case
of
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot onlybecause a
Quoted 1 line
Quoted 1 line
Quoted 1 line
trivial rename of a file breaks the client's ability to perform
I/O
Quoted 6 lines
Quoted 6 lines
by
Quoted 1 line
invalidating the filehandle. In addition, that option causes
filehandle
Quoted 1 line
aliasing (i.e. multiple filehandles pointing to the same file)
which is
Quoted 1 line
a major PITA for clients to try to manage for more or less the
same
Quoted 1 line
Quoted 1 line
Quoted 1 line
reason that it is a major PITA to try to manage these files
using
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
paths.
The discussion on volatile filehandles in RFC5661 does try to
address
Quoted 1 line
some of the above issues, but ends up concluding that you need
to
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
introduce POSIX-incompatible restrictions, such as trying to ban
renames and deletions of open files in order to make it work.None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the
sane
Quoted 20 lines
Quoted 10 lines
case.
Quoted 9 lines
That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]--
Trond Myklebust
CTO, Hammerspace Inc
4984 El Camino Real, Suite 208
Los Altos, CA 94022www.hammer.space
Signature · 6 lines
Trond Myklebust CTO, Hammerspace Inc 4984 El Camino Real, Suite 208 Los Altos, CA 94022 www.hammer.space
Quoted 4 lines · On Thu, Jan 14, 2021 at 10:08 AM [email protected] <[email protected]>…
I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.
[ Other people removed from cc, this is just a question about nfsd cleanliness ]
I missed if Trond's suggestion to at least fix up ".." to have the same filehandle as "." for the top export directory was done.
Because honestly, the whole "guessing file handles is easy" argument doesn't seem to cover the case that the client just does something wrong _by_mistake_, and this ends up then exposing the server unnecessarily that way.
It's one thing if you have an actively malicious client that is controlled by an attacker and that then makes up its own file handles.
It's another thing if you have a (benign) client that can be fooled to access files on the server that it shouldn't have.
So I think that from a pure cleanliness standpoint, the server shouldn't give the client a file handle that the client mustn't actually ever use! It's just a recipe for "oops, I didn't mean to do something bad, but by mistake..."
Hmm?
Linus
Quoted 6 lines · Linus Torvalds
Quoted 1 line · On Thu, Jan 14, 2021 at 10:08 AM [email protected] <[email protected]>…
Quoted 8 lines
Quoted 3 lines
I dug around a bit and couldn't find the idea of using filehandle guessing plus mountd's following of symlinks to get access to other filesystems. That's kind of interesting.
[ Other people removed from cc, this is just a question about nfsd cleanliness ]
I missed if Trond's suggestion to at least fix up ".." to have the same filehandle as "." for the top export directory was done.
If I understand your question correctly... there is a commit in linux-next that simply doesn't return any filehandle for ".." in the root directory.
I intend to send you a PR after a few more days of soak time, unless you'd like me to send it now.
--
Chuck Lever
Quoted 4 lines · Chuck Lever
If I understand your question correctly... there is a commit in linux-next that simply doesn't return any filehandle for ".." in the root directory.
Great.
Quoted 2 lines
I intend to send you a PR after a few more days of soak time, unless you'd like me to send it now.
No, no hurry. I was more just checking that Trond's suggestion didn't get lost in the discussion about guessing file handles.
Linus
hello,
I want to consult you on what is the original intention of designing subtree_check and whether it is to solve the 'I want to export a subtree of a filesystem' problem.
As far as I know, when opening subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry != exp->ex_path.dentry && !IS_ROOT(tdentry))' in nfsd_acceptable can work well to Intercept handles beyond the export point.
This seems to delete code as follows in nfsfh.c could solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags & NFSEXP_NOSUBTREECHECK)
return 1;
Or replace by follow:
if (exp->ex_path.dentry == exp->vfs_mount->mnt_root) return 1;
When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like subtree_check to try to verify the file handle.
I may not understand your design ideas.
Yours sincerely,
Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Quoted 20 lines · 吴异
I want to consult you on what is the original intention of designing
subtree_checkand whether it is to solve the 'I want to export a subtree of a filesystem' problem.As far as I know, when opening
subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry !=exp->ex_path.dentry&& !IS_ROOT(tdentry))' innfsd_acceptablecan work well to Intercept handles beyond the export point.This seems to delete code as follows in
nfsfh.ccould solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags&NFSEXP_NOSUBTREECHECK)
return 1;Or replace by follow:
if (exp->ex_path.dentry==exp->vfs_mount->mnt_root) return 1;
Quoted 5 lines
When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like
subtree_checkto try to verify the file handle.
Sorry, I honestly don't understand the question.
If you have a specific proposal, perhaps you could send a patch.
--b.
Quoted 111 lines
I may not understand your design ideas.
Yours sincerely,
Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
My patch is below:
/fs/nfsd/nfsfh.c
nfsd_acceptable(void expv,struct dentry *dentry)
{
- if(exp->ex_flags & NFSEXP_NOSUBTREEXHECK)
- return 1;
+ if(is_root_export(exp))
+ return 1;
+ /* If not subdirectory export, accept anything*/
}
Quoted 145 lines · 在 2021年1月19日星期二,[email protected] <[email protected]> 写道:
Quoted 20 lines · 吴异
I want to consult you on what is the original intention of designing
subtree_checkand whether it is to solve the 'I want to export a subtree of a filesystem' problem.As far as I know, when opening
subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry !=exp->ex_path.dentry&& !IS_ROOT(tdentry))' innfsd_acceptablecan work well to Intercept handles beyond the export point.This seems to delete code as follows in
nfsfh.ccould solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags&NFSEXP_NOSUBTREECHECK)
return 1;Or replace by follow:
if (exp->ex_path.dentry==exp->vfs_mount->mnt_root) return 1;Quoted 5 lines
When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like
subtree_checkto try to verify the file handle.Sorry, I honestly don't understand the question.
If you have a specific proposal, perhaps you could send a patch.
--b.
Quoted 111 lines
I may not understand your design ideas.
Yours sincerely,
Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Review
Quoted 10 lines · 吴异
My patch is below:
/fs/nfsd/nfsfh.c
nfsd_acceptable(void expv,struct dentry *dentry)
{(quoted diff) +2
-2- if(exp->ex_flags & NFSEXP_NOSUBTREEXHECK) - return 1; + if(is_root_export(exp)) + return 1;
So you end up doing the SUBTREECHECK checks in cases where the filehandle doesn't actually have any parent information. That means we may be stuck with no way to connect the file back up the export point, so this check will fail even though the file may be under the export point. So users probably get spurious ESTALE errors.
To reproduce you probably need to do something like open a file on the client, reboot the server, then try to access the open file again. The dentry cache will no longer know how to look up the parents after the reboot.
--b.
Quoted 154 lines
+ /* If not subdirectory export, accept anything*/
}
Quoted 145 lines · 在 2021年1月19日星期二,[email protected] <[email protected]> 写道:
Quoted 20 lines · 吴异
I want to consult you on what is the original intention of designing
subtree_checkand whether it is to solve the 'I want to export a subtree of a filesystem' problem.As far as I know, when opening
subtree_check, the folder's file handle does not contain the inode information of its parent directory and
'while (tdentry !=exp->ex_path.dentry&& !IS_ROOT(tdentry))' innfsd_acceptablecan work well to Intercept handles beyond the export point.This seems to delete code as follows in
nfsfh.ccould solve the 'I want to export a subtree of a filesystem' problem and ensure safety:
if (exp->ex_flags&NFSEXP_NOSUBTREECHECK)
return 1;Or replace by follow:
if (exp->ex_path.dentry==exp->vfs_mount->mnt_root) return 1;Quoted 5 lines
When I was reading the nfsd code, I was confused about whether the designer used the file system as a security boundary or an export point.Since exporting a complete file system is the safest, why not directly prohibit unsafe practices, but add code like
subtree_checkto try to verify the file handle.Sorry, I honestly don't understand the question.
If you have a specific proposal, perhaps you could send a patch.
--b.
Quoted 111 lines
I may not understand your design ideas.
Yours sincerely,
Quoted 105 lines · Trond Myklebust <[email protected]> 于2021年1月13日周三 上午12:53写道:
Quoted 66 lines · J. Bruce Fields
Quoted 6 lines · 吴异
Telling users how to configure the exported file system in the most secure
way does
mitigate the problem to some extent, but this does not seem to address the
security risks posed byno_subtree_check in the code. In myQuoted 8 lines
opinion,when
the generated filehandle does not contain the inode information of the
parent directory,thenfsd_acceptablefunction can also recursively determine whether the request file exceeds the export path dentry.Enablingsubtree_checkto add parent directory information only brings some troubles.Filesystems don't necessarily provide us with an efficient way to find
parent directories from any given file. (And note a single file may have multiple parent directories.)(I do wonder if we could do better in the directory case, though. We already reconnect directories all the way back up to the root.)
Quoted 19 lines
I have a bold idea, why not directly remove the file handle modification in
subtree_check, and then normalize the judgment of whether dentry exceeds
the export point directory innfsd_acceptable(line 38 to 54 in /fs/nfsd/nfsfh.c) .As far as I understand it, the reason why
subtree_checkis not turned on by
default is that it will cause problems when reading and writing files,
rather than it wastes more time whennfsd_acceptable.In short,I think it's open to question whether the security of the system
depends on the user's complete correct configuration(the system does not
prohibit the export of a subdirectory).Quoted 8 lines
Enabling
subtree_checkto add parent directoryinformation only brings
some troubles.In short,I think it's open to question whether the security of the system depends on the user's complete correct configuration(the system
does not prohibit the export of a subdirectory).I'd love to replace the export interface by one that prohibited subdirectory exports (or at least made it more obvious where they're being used.)
But given the interface we already have, that would be a disruptive and
time-consuming change.Another approach is to add more entropy to filehandles so they're harder
to guess; see e.g.:https://www.fsl.cs.stonybrook.edu/docs/nfscrack-tr/index.html
Quoted 8 lines
In the end none of these change the fact that a filehandle has an infinite lifetime, so once it's leaked, there's nothing you can do. The
authors suggest NFSv4 volatile filehandles as a solution to that problem, but I don't think they've thought through the obstacles to making volatile filehandles work.--b.
The point is that there is no good solution to the 'I want to export a subtree of a filesystem' problem, and so it is plainly wrong to try to make a default of those solutions, which break the one sane case of exporting the whole filesystem.
Just a reminder that we kicked out
subtree_checknot only because a trivial rename of a file breaks the client's ability to perform I/O by invalidating the filehandle. In addition, that option causes filehandle aliasing (i.e. multiple filehandles pointing to the same file) which is a major PITA for clients to try to manage for more or less the same reason that it is a major PITA to try to manage these files using paths.The discussion on volatile filehandles in RFC5661 does try to address some of the above issues, but ends up concluding that you need to introduce POSIX-incompatible restrictions, such as trying to ban renames and deletions of open files in order to make it work.
None of these compromises are necessary if you export a whole filesystem (or a hierarchy of whole filesystems). That's the sane case. That's the one that people should default to using.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]