NFSv3 and xprtsec policies

DISCUSSION Scott Mayhew 4 participants 11 messages May 2, 2024 – May 6, 2024

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Scott Mayhew

author

2024-05-02 15:54 UTC #0 lore raw

Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).

Here's an example using machines from the recent Bakeathon.

Mounting a server with TLS enabled:

# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt

Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":

# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt

Yet a v3 mount without "xprtsec=tls" works:

# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt

and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":

# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0

Even though the filesystem is mounted, the client can't do anything with it:

# ls /mnt
ls: cannot open directory '/mnt': Permission denied

When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).

Perhaps we should be doing something similar with xprtsec policies? Should there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?

If not, this behavior should at least be documented in the man pages.

-Scott

Chuck Lever III

2024-05-02 16:15 UTC #1 lore raw

Quoted 39 lines · Scott Mayhew

Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied

Quoted 7 lines

When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?

The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.

Quoted 3 lines

Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?

No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.

(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).

Quoted 1 line

If not, this behavior should at least be documented in the man pages.

"crossmnt", and it's kin "nohide", are explained in exports(5).

--
Chuck Lever

Scott Mayhew

author

2024-05-02 17:37 UTC #2 lore raw

Quoted 45 lines · Chuck Lever III

Quoted 45 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).

Quoted 6 lines

Quoted 1 line
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.

I was mounting your server though :)

Quoted 11 lines

Quoted 5 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).

I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.

Quoted 5 lines

Quoted 3 lines
If not, this behavior should at least be documented in the man pages.
"crossmnt", and it's kin "nohide", are explained in exports(5).

rpc.mountd doesn't do any access checking based on xprtsec policies on the export (or krb5 pseudo-flavors, for that matter), so I don't see how "crossmount" or "nohide" would have any effect here.

-Scott

Quoted 6 lines

--
Chuck Lever

Chuck Lever III

2024-05-02 17:52 UTC #3 lore raw

Quoted 49 lines · Scott Mayhew

Quoted 43 lines · Chuck Lever III
Quoted 43 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).

Quoted 8 lines

Quoted 6 lines
Quoted 1 line
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
I was mounting your server though :)

OK, then not the same bug that Olga found last year.

We should find out what FreeBSD does in this case.

Quoted 10 lines

Quoted 9 lines
Quoted 3 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).

Quoted 7 lines

I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.

RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.

NFSv3 mount failover logic is still open for discussion (ie, incomplete).

Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.

Quoted 7 lines

Quoted 3 lines
Quoted 1 line
If not, this behavior should at least be documented in the man pages.
"crossmnt", and it's kin "nohide", are explained in exports(5).
rpc.mountd doesn't do any access checking based on xprtsec policies on the export (or krb5 pseudo-flavors, for that matter), so I don't see how "crossmount" or "nohide" would have any effect here.

No, they don't, you are correct.

--
Chuck Lever

Scott Mayhew

author

2024-05-02 18:51 UTC #4 lore raw

Quoted 56 lines · Chuck Lever III

Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.

Quoted 5 lines

Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.

I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.

Quoted 25 lines

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.

Quoted 7 lines

NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.

Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.

-Scott

Quoted 17 lines

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
If not, this behavior should at least be documented in the man pages.
"crossmnt", and it's kin "nohide", are explained in exports(5).
rpc.mountd doesn't do any access checking based on xprtsec policies on the export (or krb5 pseudo-flavors, for that matter), so I don't see how "crossmount" or "nohide" would have any effect here.
No, they don't, you are correct.
--
Chuck Lever

Jeffrey Layton

maintainer

2024-05-02 19:25 UTC #5 lore raw

Quoted 100 lines · Scott Mayhew

Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.
Quoted 25 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.

Quoted 8 lines

Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.

That sounds reasonable if it can be made to work.

One could argue that to properly implement NFSv3 over TLS, that the sidecar protocols (including mountd) should be TLS-enabled as well. If that were the case, then we could just make the mount error out when the TLS handshake doesn't work.

That _is_ a bit draconian though, and I don't see anyone lining up to do the mountd work anytime soon.

Quoted 19 lines

Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
If not, this behavior should at least be documented in the man pages.
"crossmnt", and it's kin "nohide", are explained in exports(5).
rpc.mountd doesn't do any access checking based on xprtsec policies on the export (or krb5 pseudo-flavors, for that matter), so I don't see how "crossmount" or "nohide" would have any effect here.
No, they don't, you are correct.
--
Chuck Lever

Signature · 1 line

Jeffrey Layton <[email protected]>

Chuck Lever

2024-05-03 18:44 UTC #6 lore raw

In reply to Scott Mayhew

Quoted 100 lines · Scott Mayhew

Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.
Quoted 25 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.

Quoted 7 lines

Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.

If Linux is the only implementation of NFSv3 with TLS so far, then we have some latitude for innovation.

I would like to hear from the client maintainers about what they would prefer the client user experience to look like. Then NFSD's behavior can be adjusted to accommodate.

In this case, Steve would have to sign off on an rpc.mountd change to return AUTH_TLS in the auth_flavors list.

Signature · 1 line

Chuck Lever

Scott Mayhew

author

2024-05-03 20:53 UTC #7 lore raw

Quoted 112 lines · Chuck Lever

Quoted 100 lines · Scott Mayhew
Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.
Quoted 25 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.
Quoted 7 lines
Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.
If Linux is the only implementation of NFSv3 with TLS so far, then we have some latitude for innovation.

Quoted 6 lines

I would like to hear from the client maintainers about what they would prefer the client user experience to look like. Then NFSD's behavior can be adjusted to accommodate.
In this case, Steve would have to sign off on an rpc.mountd change to return AUTH_TLS in the auth_flavors list.

Maybe instead of messing with the auth_flavors list, we just have the client check the status from the PATHCONF operation and leave it at that.

---8<---

fs/nfs/client.c +4 -2

index 44eca51b2808..09d28dae0f06 100644
@@ -867,8 +867,10 @@ static int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, str
 		pathinfo.fattr = fattr;
 		nfs_fattr_init(fattr);
 
-		if (clp->rpc_ops->pathconf(server, mntfh, &pathinfo) >= 0)
-			server->namelen = pathinfo.max_namelen;
+		error = clp->rpc_ops->pathconf(server, mntfh, &pathinfo);
+		if (error < 0)
+			return error;
+		server->namelen = pathinfo.max_namelen;
 	}
 
 	if (clp->rpc_ops->discover_trunking != NULL &&

---8<---

That's sufficient to make the mount fail with EACCES, which should be enough of a clue for someone that they should compare the client's mount options with the server's export options.

[root@rawhide-client ~]# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt mount.nfs: access denied by server while mounting oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls

-Scott

Quoted 5 lines

Signature · 2 lines
-- 
Chuck Lever

Rick Macklem

2024-05-03 22:31 UTC #8 lore raw

In reply to Chuck Lever

Quoted 60 lines · Chuck Lever

Quoted 58 lines · Scott Mayhew
Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.

Quoted 8 lines

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.

If you want me to leave the FreeBSD server up on tailscale configured to allow NFSv3/RPC-with-TLS mounts, just email.

I did a quick test using the FreeBSD client and the mount fails, but I know that failure happens when it tries to access the server file system in the kernel.
The replies fail with RPC Msg_denied.
(I've attached a packet trace, in case you are interested.)

I haven't done anything with the userspace mountd daemon, which means it will allow a Mount protocol RPC without TLS and does not return any different info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard, but I have tried hard to avoid adding RPC-with-TLS to the userspace RPC libraries.
(Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)

Quoted 42 lines

Quoted 34 lines
Quoted 26 lines
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
Quoted 5 lines
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.
Quoted 7 lines
Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.

Quoted 2 lines

If Linux is the only implementation of NFSv3 with TLS so far, then we have some latitude for innovation.

At this point, FreeBSD can change to whatever you guys think works best, although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.

Have fun with it, rick

Quoted 12 lines

I would like to hear from the client maintainers about what they would prefer the client user experience to look like. Then NFSD's behavior can be adjusted to accommodate.
In this case, Steve would have to sign off on an rpc.mountd change to return AUTH_TLS in the auth_flavors list.
--
Chuck Lever

Scott Mayhew

author

2024-05-06 16:47 UTC #9 lore raw

Quoted 69 lines · Rick Macklem

Quoted 60 lines · Chuck Lever
Quoted 58 lines · Scott Mayhew
Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.

Quoted 8 lines

If you want me to leave the FreeBSD server up on tailscale configured to allow NFSv3/RPC-with-TLS mounts, just email.
I did a quick test using the FreeBSD client and the mount fails, but I know that failure happens when it tries to access the server file system in the kernel.
The replies fail with RPC Msg_denied.
(I've attached a packet trace, in case you are interested.)

Thanks, Rick. Looking at that trace, freebsd is rejecting the GETATTR and FSINFO requests at the RPC layer, whereas Linux knfsd isn't.

Digging a little further, both nfsd3_proc_fsinfo() and nfsd3_proc_getattr() are calling fh_verify() with a flag called NFSD_MAY_BYPASS_GSS_ON_ROOT

---8<---

        resp->status = fh_verify(rqstp, &resp->fh, 0,
                                 NFSD_MAY_NOP | NFSD_MAY_BYPASS_GSS_ON_ROOT);

---8<---

and fh_verify() has

---8<---

        /*
         * Clients may expect to be able to use auth_sys during mount,
         * even if they use gss for everything else; see section 2.3.2
         * of rfc 2623.
         */
        if (access & NFSD_MAY_BYPASS_GSS_ON_ROOT
                        && exp->ex_path.dentry == dentry)
                goto skip_pseudoflavor_check;

        error = check_nfsd_access(exp, rqstp);

---8<---

so it's skipping check_nfsd_access(), which is where the xprtsec policy checking occurs.

Looking at RFC 2623 section 2.3.2, it basically says a server can choose to not require authentication for GETATTR and FSINFO. Obviously, RFC 2623 predates RPC-with-TLS by 20+ years, so there's nothing there to dictate what to do here... but as Chuck reiterated, TLS isn't a security flavor so maybe knfsd shouldn't be bypassing those xprtsec policy checks at all.

-Scott

Quoted 69 lines

I haven't done anything with the userspace mountd daemon, which means it will allow a Mount protocol RPC without TLS and does not return any different info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard, but I have tried hard to avoid adding RPC-with-TLS to the userspace RPC libraries.
(Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)
Quoted 42 lines
Quoted 34 lines
Quoted 26 lines
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
Quoted 5 lines
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.
Quoted 7 lines
Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.
Quoted 2 lines
If Linux is the only implementation of NFSv3 with TLS so far, then we have some latitude for innovation.
At this point, FreeBSD can change to whatever you guys think works best, although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.
Have fun with it, rick
Quoted 12 lines
I would like to hear from the client maintainers about what they would prefer the client user experience to look like. Then NFSD's behavior can be adjusted to accommodate.
In this case, Steve would have to sign off on an rpc.mountd change to return AUTH_TLS in the auth_flavors list.
--
Chuck Lever

Rick Macklem

2024-05-06 22:41 UTC #10 lore raw

Quoted 105 lines · Scott Mayhew

Quoted 69 lines · Rick Macklem
Quoted 60 lines · Chuck Lever
Quoted 58 lines · Scott Mayhew
Quoted 56 lines · Chuck Lever III
Quoted 56 lines · Scott Mayhew
Quoted 49 lines · Chuck Lever III
Quoted 37 lines · Scott Mayhew
Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an NFSv3 client can mount with "xprtsec=none" a filesystem exported with "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
Quoted 7 lines
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the mount request and returns -EACCES if no match is found (see nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3, the parent export needs the "crossmnt" export option in order for NFSv3 to behave like NFSv4 in this regard, although I could have missed something.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I don't think he was exporting v3 anyway.
Quoted 8 lines
If you want me to leave the FreeBSD server up on tailscale configured to allow NFSv3/RPC-with-TLS mounts, just email.
I did a quick test using the FreeBSD client and the mount fails, but I know that failure happens when it tries to access the server file system in the kernel.
The replies fail with RPC Msg_denied.
(I've attached a packet trace, in case you are interested.)
Thanks, Rick. Looking at that trace, freebsd is rejecting the GETATTR and FSINFO requests at the RPC layer, whereas Linux knfsd isn't.
Digging a little further, both nfsd3_proc_fsinfo() and nfsd3_proc_getattr() are calling fh_verify() with a flag called NFSD_MAY_BYPASS_GSS_ON_ROOT
---8<---
        resp->status = fh_verify(rqstp, &resp->fh, 0,
                                 NFSD_MAY_NOP | NFSD_MAY_BYPASS_GSS_ON_ROOT);
---8<---
and fh_verify() has
---8<---
        /*
         * Clients may expect to be able to use auth_sys during mount,
         * even if they use gss for everything else; see section 2.3.2
         * of rfc 2623.
         */
        if (access & NFSD_MAY_BYPASS_GSS_ON_ROOT
                        && exp->ex_path.dentry == dentry)
                goto skip_pseudoflavor_check;

        error = check_nfsd_access(exp, rqstp);

Quoted 7 lines

---8<---
so it's skipping check_nfsd_access(), which is where the xprtsec policy checking occurs.
Looking at RFC 2623 section 2.3.2, it basically says a server can choose to not require authentication for GETATTR and FSINFO.

My guess would be that this was done to allow root to do a mount without having a TGT. I don't think the concept of a host/fqdn@realm keytab entry was invented until NFSv4 came along.

Quoted 5 lines

Obviously, RFC
2623 predates RPC-with-TLS by 20+ years, so there's nothing there to dictate what to do here... but as Chuck reiterated, TLS isn't a security flavor so maybe knfsd shouldn't be bypassing those xprtsec policy checks at all.

I'd agree with Chuck and do not see why a mount being done by root cannot do GETATTR/FSINFO after doing a TLS handshake.
But, obviously, it is up to you guys.

Have fun with it, rick

Quoted 75 lines

-Scott
Quoted 69 lines
I haven't done anything with the userspace mountd daemon, which means it will allow a Mount protocol RPC without TLS and does not return any different info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard, but I have tried hard to avoid adding RPC-with-TLS to the userspace RPC libraries.
(Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)
Quoted 42 lines
Quoted 34 lines
Quoted 26 lines
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for pseudo-flavors corresponding to xprtsec policies?
Quoted 5 lines
No. Transport-layer security is not an RPC security flavor or pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO, but it was rejected).
Quoted 7 lines
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if it supports RPC-with-TLS. At all other times, the client uses one of the normal, legitimate flavors. It does not represent a security flavor that can be used during regular operation.
Quoted 7 lines
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors list? I think clients that don't recognize it should ignore it, but I'm not sure. What should a client do if it sees that flavor in the list? It's not one that can be used for any other procedure than a NULL RPC.
Quoted 7 lines
Maybe? After the client gets the filehandle it's calling FSINFO and PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't checking for a negative return code from the PATHCONF operation. If it did, it could maybe use the -EACCES coupled with the knowledge that the server had RPC_AUTH_TLS enabled to emit an error message saying to check the xprtsec policies (but I don't think that would be as definitive as what I had in mind) and to fail the mount.
Quoted 2 lines
If Linux is the only implementation of NFSv3 with TLS so far, then we have some latitude for innovation.
At this point, FreeBSD can change to whatever you guys think works best, although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.
Have fun with it, rick
Quoted 12 lines
I would like to hear from the client maintainers about what they would prefer the client user experience to look like. Then NFSD's behavior can be adjusted to accommodate.
In this case, Steve would have to sign off on an rpc.mountd change to return AUTH_TLS in the auth_flavors list.
--
Chuck Lever