NFSD: Support write delegations for pNFS LAYOUT operations
Subsystems (2)
[email protected]
maintainerRFC PATCH 1/1 NFSD: Support write delegations for pNFS LAYOUT operations
From: Chuck Lever <[email protected]>
I noticed LAYOUTGET(LAYOUTIOMODE4_RW) returning NFS4ERR_ACCESS unexpectedly. The NFS client had created a file with mode 0444, and the server had returned a write delegation on the OPEN(CREATE). The client was requesting a RW layout using the write delegation stateid so that it could flush file modifications.
This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because fh_verify() was doing a permission check as part of verifying the FH. It uses the loga_iomode value to specify the @accmode argument. fh_verify(MAY_WRITE) on a file whose mode is 0444 fails with -EACCES.
RFC 8881 Section 18.43.3 states:
Quoted 2 lines
The use of the
loga_iomodefield depends upon the layout type, but should reflect the client's data access intent.
Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Appropriate permission checking is done when the client acquires the stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.
Cc: Christoph Hellwig <[email protected]>
X-Cc: [email protected] # v6.6+
--- fs/nfsd/nfs4proc.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)
fs/nfsd/nfs4proc.c +3 -9
@@ -2269,23 +2269,17 @@ nfsd4_layoutget(struct svc_rqst *rqstp,const struct nfsd4_layout_ops *ops;struct nfs4_layout_stateid *ls;__be32 nfserr; -int accmode =NFSD_MAY_READ_IF_EXEC ; + nfserr = nfserr_badiomode;switch (lgp->lg_seg.iomode) {case IOMODE_READ : - accmode |=NFSD_MAY_READ ; -break ;case IOMODE_RW : - accmode |=NFSD_MAY_READ |NFSD_MAY_WRITE ;break ;default : -dprintk ("%s: invalid iomode %d\n" , - __func__, lgp->lg_seg.iomode); - nfserr = nfserr_badiomode; goto out; } - nfserr =fh_verify (rqstp, current_fh,0 , accmode); + nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_NOP );if (nfserr) goto out;
@@ -2359,7 +2353,7 @@ nfsd4_layoutcommit(struct svc_rqst *rqstp,struct nfs4_layout_stateid *ls;__be32 nfserr; - nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_WRITE ); + nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_NOP );if (nfserr) goto out;
Signature · 1 line
2.45.1
Quoted 25 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and the server had returned a write delegation on the OPEN(CREATE). The client was requesting a RW layout using the write delegation stateid so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because
fh_verify()was doing a permission check as part of verifying the FH. It uses theloga_iomodevalue to specify the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode is 0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 2 lines
The use of the
loga_iomodefield depends upon the layout type, but should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Quoted 4 lines
Appropriate permission checking is done when the client acquires the stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.
Hmm... I'm not seeing any checking or enforcement of the EXCHGID4_FLAG_BIND_PRINC_STATEID flag in knfsd. Doesn't that mean that READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?
I'm not saying that the return of an ACCESS error is correct here, since the file was created using this credential, and so the caller should benefit from having owner privileges.
However the issue is that knfsd should be either checking that the credential is correct w.r.t. the stateid (if EXCHGID4_FLAG_BIND_PRINC_STATEID is set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions (if EXCHGID4_FLAG_BIND_PRINC_STATEID is not set or the stateid is a special stateid).
Quoted 47 lines
Cc: Christoph Hellwig <[email protected]>
X-Cc: [email protected] # v6.6+Signed-off-by Chuck Lever <[email protected]>---
fs/nfsd/nfs4proc.c | 12 +++---------1 file changed, 3 insertions(+), 9 deletions(-)fs/nfsd/nfs4proc.c +0
-1@@ -2269,23 +2269,17 @@ nfsd4_layoutget(struct svc_rqst *rqstp, const struct nfsd4_layout_ops *ops; struct nfs4_layout_stateid *ls; __be32 nfserr; -int accmode =NFSD_MAY_READ_IF_EXEC ;fs/nfsd/nfs4proc.c +1
-0+ nfserr = nfserr_badiomode;switch (
lgp->lg_seg.iomode) {
caseIOMODE_READ:fs/nfsd/nfs4proc.c +0
-2- accmode |=NFSD_MAY_READ ; -break ;case
IOMODE_RW:fs/nfsd/nfs4proc.c +0
-1- accmode |=NFSD_MAY_READ |NFSD_MAY_WRITE ;break;
default:fs/nfsd/nfs4proc.c +0
-3-dprintk ("%s: invalid iomode %d\n" , - __func__, lgp->lg_seg.iomode); - nfserr = nfserr_badiomode;goto out;
}fs/nfsd/nfs4proc.c +1
-1- nfserr =fh_verify (rqstp, current_fh,0 , accmode); + nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_NOP );if (nfserr)
goto out;fs/nfsd/nfs4proc.c +0
-0@@ -2359,7 +2353,7 @@ nfsd4_layoutcommit(struct svc_rqst *rqstp, struct nfs4_layout_stateid *ls; __be32 nfserr;fs/nfsd/nfs4proc.c +1
-1- nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_WRITE ); + nfserr =fh_verify (rqstp, current_fh,0 ,NFSD_MAY_NOP );if (nfserr)
goto out;
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 26 lines · Trond Myklebust
Quoted 25 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and the server had returned a write delegation on the OPEN(CREATE). The client was requesting a RW layout using the write delegation stateid so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because
fh_verify()was doing a permission check as part of verifying the FH. It uses theloga_iomodevalue to specify the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode is 0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 2 lines
The use of the
loga_iomodefield depends upon the layout type, but should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Quoted 7 lines
Quoted 4 lines
Appropriate permission checking is done when the client acquires the stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.
Hmm... I'm not seeing any checking or enforcement of the
EXCHGID4_FLAG_BIND_PRINC_STATEIDflag in knfsd.
I appreciate your review!
I see that BIND_PRINC_STATEID is not set by NFSD. RFC 8881 Section 18.16.4 says:
Quoted 6 lines
Note that if the client ID was not created with the
EXCHGID4_FLAG_BIND_PRINC_STATEIDcapability set in the reply toEXCHANGE_ID, then the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself, and the server is REQUIRED to perform access checking on the READs and WRITEs themselves.
Trond:
Quoted 3 lines
Doesn't that mean that
READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?
Every NFSv4 READ and WRITE calls nfs4_preprocess_stateid_op(), and nfs4_preprocess_stateid_op() calls nfsd_permission() (in nfs4_check_file()). Seems like we're covered.
Trond:
Quoted 3 lines
I'm not saying that the return of an ACCESS error is correct here, since the file was created using this credential, and so the caller should benefit from having owner privileges.
The alternative is to preserve the accmode logic but instead add the NFSD_MAY_OWNER_OVERRIDE flag, me thinks, which is not objectionable.
Trond:
Quoted 6 lines
However the issue is that knfsd should be either checking that the credential is correct w.r.t. the stateid (if
EXCHGID4_FLAG_BIND_PRINC_STATEIDis set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions (ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis not set or the stateid is a special stateid).
But LAYOUTGET is not a READ or WRITE. I don't see language that requires stateid / credential checking for it, but it's always possible I might have missed something.
Also, RFC 5663 has nothing to say about BIND_PRINC_STATEID. Further, I'm not sure how a SCSI READ or WRITE can check credentials as NFS stateids are not presented to SCSI/iSCSI targets.
Likewise, how would this impact a flexfile layout that targets an NFSv3 DS?
I think NFSD is checking stateids used for NFSv4 READ and WRITE as needed, but help me understand how BIND_PRINC_STATEID is applicable to pNFS block layouts...?
Signature · 1 line
Chuck Lever
Quoted 92 lines · Chuck Lever
Quoted 33 lines · Trond Myklebust
Quoted 32 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and
the server had returned a write delegation on the OPEN(CREATE). The
client was requesting a RW layout using the write delegation stateid
so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because
fh_verify()was doing a permission check as
part of verifying the FH. It uses theloga_iomodevalue to specify
the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode is
0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 3 lines
The use of the
loga_iomodefield depends upon the layout type, but
should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Quoted 8 lines
Quoted 5 lines
Appropriate permission checking is done when the client acquires the
stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.Hmm... I'm not seeing any checking or enforcement of the
EXCHGID4_FLAG_BIND_PRINC_STATEIDflag in knfsd.I appreciate your review!
I see that
BIND_PRINC_STATEIDis not set by NFSD. RFC 8881 Section 18.16.4 says:Quoted 6 lines
Note that if the client ID was not created with the
EXCHGID4_FLAG_BIND_PRINC_STATEIDcapability set in the reply toEXCHANGE_ID, then the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself, and the server is REQUIRED to perform access checking on the READs and WRITEs themselves.Trond:
Quoted 3 lines
Doesn't that mean that
READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?Every NFSv4 READ and WRITE calls
nfs4_preprocess_stateid_op(), andnfs4_preprocess_stateid_op()callsnfsd_permission()(innfs4_check_file()). Seems like we're covered.Trond:
Quoted 3 lines
I'm not saying that the return of an ACCESS error is correct here, since the file was created using this credential, and so the caller should benefit from having owner privileges.
The alternative is to preserve the accmode logic but instead add the
NFSD_MAY_OWNER_OVERRIDEflag, me thinks, which is not objectionable.Trond:
Quoted 8 lines
However the issue is that knfsd should be either checking that the credential is correct w.r.t. the stateid (if
EXCHGID4_FLAG_BIND_PRINC_STATEIDis set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions (ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis not set or the stateid is a special
stateid).But LAYOUTGET is not a READ or WRITE. I don't see language that requires stateid / credential checking for it, but it's always possible I might have missed something.
Also, RFC 5663 has nothing to say about
BIND_PRINC_STATEID. Further, I'm not sure how a SCSI READ or WRITE can check credentials as NFS stateids are not presented to SCSI/iSCSI targets.
Quoted 8 lines
Likewise, how would this impact a flexfile layout that targets an NFSv3 DS?
I think NFSD is checking stateids used for NFSv4 READ and WRITE as needed, but help me understand how
BIND_PRINC_STATEIDis applicable to pNFS block layouts...?
If you look at Section 15.2, then NFS4ERR_ACCESS is definitely listed as a valid error for LAYOUTGET (in fact, the very first in the list).
Furthermore, please see the following blanket statement in RFC8881, section 12.5.1.:
"Layouts are provided to NFSv4.1 clients, and user access still follows the rules of the protocol as if they did not exist."
While you can argue that the above language is not normative, because it doesn't use the official IETF "MUST" / "MUST NOT" /..., it clearly does declare an intention to ensure that pNFS not be allowed to weaken the protocol.
So my point would be that if LAYOUTGET is the only time where a proper credential check can occur, then it needs to happen there. Otherwise, your implementation is responsible for ensuring that it happens at some other time, in order to ensure that user access follows the rules of the protocol.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 134 lines · Trond Myklebust
Quoted 109 lines · Chuck Lever
Quoted 49 lines · Trond Myklebust
Quoted 39 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and
the server had returned a write delegation on the OPEN(CREATE). The
client was requesting a RW layout using the write delegation stateid
so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I
began looking at NFSD's implementation of LAYOUTGET.The failure was because
fh_verify()was doing a permission check
as
part of verifying the FH. It uses theloga_iomodevalue to specify
the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode
is
0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 4 lines
The use of the
loga_iomodefield depends upon the layout type,
but
should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is
permitted to change returned the iomode when coalescing layouts.
It says nothing about mandating the denial of LAYOUTGET requests
due to file permission settings.Quoted 8 lines
Appropriate permission checking is done when the client acquires
the
stateid used in the LAYOUTGET operation, so remove the permission
check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid
checking instead.Quoted 2 lines
Hmm... I'm not seeing any checking or enforcement of the
EXCHGID4_FLAG_BIND_PRINC_STATEIDflag in knfsd.I appreciate your review!
I see that
BIND_PRINC_STATEIDis not set by NFSD. RFC 8881 Section 18.16.4 says:Quoted 6 lines
Note that if the client ID was not created with the
EXCHGID4_FLAG_BIND_PRINC_STATEIDcapability set in the reply toEXCHANGE_ID, then the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself, and the server is REQUIRED to perform access checking on the READs and WRITEs themselves.Trond:
Quoted 3 lines
Doesn't that mean that
READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?Every NFSv4 READ and WRITE calls
nfs4_preprocess_stateid_op(), andnfs4_preprocess_stateid_op()callsnfsd_permission()(innfs4_check_file()). Seems like we're covered.Trond:
Quoted 5 lines
I'm not saying that the return of an ACCESS error is correct here,
since the file was created using this credential, and so the caller
should benefit from having owner privileges.The alternative is to preserve the accmode logic but instead add the
NFSD_MAY_OWNER_OVERRIDEflag, me thinks, which is not objectionable.Trond:
Quoted 10 lines
However the issue is that knfsd should be either checking that the
credential is correct w.r.t. the stateid (ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions
(ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis not set or the stateid is a special
stateid).But LAYOUTGET is not a READ or WRITE. I don't see language that requires stateid / credential checking for it, but it's always possible I might have missed something.
Also, RFC 5663 has nothing to say about
BIND_PRINC_STATEID. Further,
I'm not sure how a SCSI READ or WRITE can check credentials as NFS stateids are not presented to SCSI/iSCSI targets.Quoted 8 lines
Likewise, how would this impact a flexfile layout that targets an NFSv3 DS?
I think NFSD is checking stateids used for NFSv4 READ and WRITE as needed, but help me understand how
BIND_PRINC_STATEIDis applicable to pNFS block layouts...?If you look at Section 15.2, then
NFS4ERR_ACCESSis definitely listed as a valid error for LAYOUTGET (in fact, the very first in the list).Furthermore, please see the following blanket statement in RFC8881, section 12.5.1.:
"Layouts are provided to NFSv4.1 clients, and user access still follows the rules of the protocol as if they did not exist."
While you can argue that the above language is not normative, because it doesn't use the official IETF "MUST" / "MUST NOT" /..., it clearly does declare an intention to ensure that pNFS not be allowed to weaken
the protocol.
Quoted 8 lines
So my point would be that if LAYOUTGET is the only time where a proper
credential check can occur, then it needs to happen there. Otherwise, your implementation is responsible for ensuring that it happens at some
other time, in order to ensure that user access follows the rules of the protocol.
Put differently: unless you have an alternative mechanism for ensuring that the NFSv4.1 rules are followed, then
* If EXCHGID4_FLAG_BIND_PRINC_STATEID is not set, then LAYOUTGET needs to check the supplied credential against the file permissions, and a file permissions change while a layout is outstanding should almost certainly trigger a layout recall.
* If EXCHGID4_FLAG_BIND_PRINC_STATEID is set, then LAYOUTGET needs to check the credential against the one that was used to create the stateid, however it should presumably not be necessary to perform the layout recall during a file permissions change.
Signature · 2 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Tom Talpey
reviewerQuoted 116 lines · Trond Myklebust
Quoted 92 lines · Chuck Lever
Quoted 33 lines · Trond Myklebust
Quoted 32 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and
the server had returned a write delegation on the OPEN(CREATE). The
client was requesting a RW layout using the write delegation stateid
so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because
fh_verify()was doing a permission check as
part of verifying the FH. It uses theloga_iomodevalue to specify
the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode is
0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 3 lines
The use of the
loga_iomodefield depends upon the layout type, but
should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Quoted 8 lines
Quoted 5 lines
Appropriate permission checking is done when the client acquires the
stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.Hmm... I'm not seeing any checking or enforcement of the
EXCHGID4_FLAG_BIND_PRINC_STATEIDflag in knfsd.I appreciate your review!
I see that
BIND_PRINC_STATEIDis not set by NFSD. RFC 8881 Section 18.16.4 says:Quoted 6 lines
Note that if the client ID was not created with the
EXCHGID4_FLAG_BIND_PRINC_STATEIDcapability set in the reply toEXCHANGE_ID, then the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself, and the server is REQUIRED to perform access checking on the READs and WRITEs themselves.Trond:
Quoted 3 lines
Doesn't that mean that
READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?Every NFSv4 READ and WRITE calls
nfs4_preprocess_stateid_op(), andnfs4_preprocess_stateid_op()callsnfsd_permission()(innfs4_check_file()). Seems like we're covered.Trond:
Quoted 3 lines
I'm not saying that the return of an ACCESS error is correct here, since the file was created using this credential, and so the caller should benefit from having owner privileges.
The alternative is to preserve the accmode logic but instead add the
NFSD_MAY_OWNER_OVERRIDEflag, me thinks, which is not objectionable.Trond:
Quoted 8 lines
However the issue is that knfsd should be either checking that the credential is correct w.r.t. the stateid (if
EXCHGID4_FLAG_BIND_PRINC_STATEIDis set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions (ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis not set or the stateid is a special
stateid).But LAYOUTGET is not a READ or WRITE. I don't see language that requires stateid / credential checking for it, but it's always possible I might have missed something.
Also, RFC 5663 has nothing to say about
BIND_PRINC_STATEID. Further, I'm not sure how a SCSI READ or WRITE can check credentials as NFS stateids are not presented to SCSI/iSCSI targets.Quoted 8 lines
Likewise, how would this impact a flexfile layout that targets an NFSv3 DS?
I think NFSD is checking stateids used for NFSv4 READ and WRITE as needed, but help me understand how
BIND_PRINC_STATEIDis applicable to pNFS block layouts...?If you look at Section 15.2, then
NFS4ERR_ACCESSis definitely listed as a valid error for LAYOUTGET (in fact, the very first in the list).Furthermore, please see the following blanket statement in RFC8881, section 12.5.1.:
"Layouts are provided to NFSv4.1 clients, and user access still follows the rules of the protocol as if they did not exist."
While you can argue that the above language is not normative, because it doesn't use the official IETF "MUST" / "MUST NOT" /..., it clearly does declare an intention to ensure that pNFS not be allowed to weaken the protocol.
Quoted 5 lines
So my point would be that if LAYOUTGET is the only time where a proper credential check can occur, then it needs to happen there. Otherwise, your implementation is responsible for ensuring that it happens at some other time, in order to ensure that user access follows the rules of the protocol.
Absolutely agreed. The MDS needs to verify access before returning the layout, for no other reason that the DS can't perform the same verification.
Tom.
Quoted 122 lines · Tom Talpey
Quoted 101 lines · Trond Myklebust
Quoted 92 lines · Chuck Lever
Quoted 33 lines · Trond Myklebust
Quoted 32 lines · On Mon, 2024-06-10 at 11:04 -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
I noticed
LAYOUTGET(LAYOUTIOMODE4_RW)returningNFS4ERR_ACCESSunexpectedly. The NFS client had created a file with mode 0444, and
the server had returned a write delegation on the OPEN(CREATE). The
client was requesting a RW layout using the write delegation stateid
so that it could flush file modifications.This client behavior was permitted for NFSv4.1 without pNFS, so I began looking at NFSD's implementation of LAYOUTGET.
The failure was because
fh_verify()was doing a permission check as
part of verifying the FH. It uses theloga_iomodevalue to specify
the @accmode argument.fh_verify(MAY_WRITE)on a file whose mode is
0444 fails with-EACCES.RFC 8881 Section 18.43.3 states:
Quoted 3 lines
The use of the
loga_iomodefield depends upon the layout type, but
should reflect the client's data access intent.Further discussion of iomode values focuses on how the server is permitted to change returned the iomode when coalescing layouts. It says nothing about mandating the denial of LAYOUTGET requests due to file permission settings.
Quoted 8 lines
Quoted 5 lines
Appropriate permission checking is done when the client acquires the
stateid used in the LAYOUTGET operation, so remove the permission check from LAYOUTGET and LAYOUTCOMMIT, and rely on layout stateid checking instead.Hmm... I'm not seeing any checking or enforcement of the
EXCHGID4_FLAG_BIND_PRINC_STATEIDflag in knfsd.I appreciate your review!
I see that
BIND_PRINC_STATEIDis not set by NFSD. RFC 8881 Section 18.16.4 says:Quoted 6 lines
Note that if the client ID was not created with the
EXCHGID4_FLAG_BIND_PRINC_STATEIDcapability set in the reply toEXCHANGE_ID, then the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself, and the server is REQUIRED to perform access checking on the READs and WRITEs themselves.Trond:
Quoted 3 lines
Doesn't that mean that
READ and WRITE are required to check access permissions, as per RFC8881, section 13.9.2.3?Every NFSv4 READ and WRITE calls
nfs4_preprocess_stateid_op(), andnfs4_preprocess_stateid_op()callsnfsd_permission()(innfs4_check_file()). Seems like we're covered.Trond:
Quoted 3 lines
I'm not saying that the return of an ACCESS error is correct here, since the file was created using this credential, and so the caller should benefit from having owner privileges.
The alternative is to preserve the accmode logic but instead add the
NFSD_MAY_OWNER_OVERRIDEflag, me thinks, which is not objectionable.Trond:
Quoted 8 lines
However the issue is that knfsd should be either checking that the credential is correct w.r.t. the stateid (if
EXCHGID4_FLAG_BIND_PRINC_STATEIDis set and the stateid is not a special stateid) or that it is correct w.r.t. the file permissions (ifEXCHGID4_FLAG_BIND_PRINC_STATEIDis not set or the stateid is a special
stateid).But LAYOUTGET is not a READ or WRITE. I don't see language that requires stateid / credential checking for it, but it's always possible I might have missed something.
Also, RFC 5663 has nothing to say about
BIND_PRINC_STATEID. Further, I'm not sure how a SCSI READ or WRITE can check credentials as NFS stateids are not presented to SCSI/iSCSI targets.Quoted 8 lines
Likewise, how would this impact a flexfile layout that targets an NFSv3 DS?
I think NFSD is checking stateids used for NFSv4 READ and WRITE as needed, but help me understand how
BIND_PRINC_STATEIDis applicable to pNFS block layouts...?Quoted 15 lines
If you look at Section 15.2, then
NFS4ERR_ACCESSis definitely listed as a valid error for LAYOUTGET (in fact, the very first in the list). Furthermore, please see the following blanket statement in RFC8881, section 12.5.1.:
"Layouts are provided to NFSv4.1 clients, and user access still follows the rules of the protocol as if they did not exist."
While you can argue that the above language is not normative, because it doesn't use the official IETF "MUST" / "MUST NOT" /..., it clearly does declare an intention to ensure that pNFS not be allowed to weaken the protocol.
So my point would be that if LAYOUTGET is the only time where a proper credential check can occur, then it needs to happen there. Otherwise, your implementation is responsible for ensuring that it happens at some other time, in order to ensure that user access follows the rules of the protocol.
Quoted 3 lines
Absolutely agreed. The MDS needs to verify access before returning the layout, for no other reason that the DS can't perform the same verification.
I have trouble believing that something that critical has been left by the spec authors as an implied implementation quality issue.
The bulk of Sections 12.5.1 and .2 discuss how storage devices need to handle things like writes outside of a layout and other corner cases, including permission changes during I/O. Clearly the spec authors expect that DS targets will deal properly with these situations without the use of stateids.
BIND_PRINC_STATEID is never explicitly mentioned here. Instead, the OPEN, LOCK, and ACCESS operations are clearly mentioned as the points in time when file permission is inspected by the server (ie, these operations provide the main access verification check that precedes the successful retrieval of a layout).
Block layouts then use SCSI persistent reservation and layout recall to handle I/O-time access checking.
Therefore, IMO, BIND_PRINC_STATEID applies only to NFSv4 READ and WRITE, because the spec as it stands is not sane otherwise.
However, the objection is noted. I'm working on an alternate more conservative fix for the write delegation issue that motivated this patch. Stay tuned for v2.
--
Chuck Lever