Skip to contentSkip to conversation

Client security considerations for out of band I/O

DISCUSSION Benny Halevy 2 participants 3 messages May 28, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Trond, with the latest code for issuing LAYOUTGET with the right credentials we still seem to have a problem with the objects and blocks layout where the security enforcement over out-of-band I/O differs from than the one over in-band I/O.

Consider the following scenario:

file is owned by <uid1, gid1>, mode 660
process p1 successfully opens the file for RW with <uid1, gid1> (client sent OPEN) process p2 successfully opens the file for RW with <uid2, gid1> (client sent ACCESS) client gets a layout using LAYOUTGET for IOMODE_RW
the file is chmod'ed to 600

now, empirically, in-band I/O would succeed for p1 and fail for p2 (as seen on linux and some commercial servers)

for out-of-band I/O, an object-based server will fence-off the object and recall the layout to enforce the client to refresh its layout, send a LAYOUTGET, reauthorize, and get a new capability. BUT, that's not enough as the new layout and capability, would allow both p1 and p2 access to the object (as the layout is global to the client), yet we want only p1 to have access now.

How about sending ACCESS for any principal before using a newly retrieved layout at OPEN time or after the layout was revoked/reacquired to simulate the in-band behavior in a practical manner?

Note that I expect some inaccuracies in behavior even with sending ACCESS as
the linux nfs server and other commercial servers bypass permission checking for the file owner at I/O time but not for ACCESS. I believe this was done to simulate (sort of) Posix behavior that allows I/O to an open file even after changing its security attributes.

Also, do we deal correctly with LAYOUTGET failing on NFS4ERR_ACCESS?
In the example above, if the open order was reversed, LAYOUTGET would have failed for p2's creds as it doesn't have RW access to the file. That would result to reverting to the MDS and the I/O would fail on NFS4ERR_ACCESS as well, yet we'll keep trying (and failing) LAYOUTGET. Optionally, the client could try other creds that opened the file. If the first process to open the file closes it, should we use different creds for LAYOUTGET? With the latest implementation we keep the first opener creds referenced until we return the whole layout, right?

Regards,

Benny

#1 lore raw
Quoted 13 lines · Benny Halevy

Trond, with the latest code for issuing LAYOUTGET with the right credentials we still seem to have a problem with the objects and blocks layout where the security enforcement over out-of-band I/O differs from than the one over in-band I/O.

Consider the following scenario:

file is owned by <uid1, gid1>, mode 660
process p1 successfully opens the file for RW with <uid1, gid1> (client sent OPEN) process p2 successfully opens the file for RW with <uid2, gid1> (client sent ACCESS) client gets a layout using LAYOUTGET for IOMODE_RW
the file is chmod'ed to 600

Quoted 8 lines

now, empirically, in-band I/O would succeed for p1 and fail for p2 (as seen on linux and some commercial servers)

for out-of-band I/O, an object-based server will fence-off the object and recall the layout to enforce the client to refresh its layout, send a LAYOUTGET, reauthorize, and get a new capability. BUT, that's not enough as the new layout and capability, would allow both p1 and p2 access to the object (as the layout is global to the client), yet we want only p1 to have access now.

I don't understand why you think this is related to the LAYOUTGET credential change. The only difference that the credential change brings is to the case where the client doesn't hold a layout segment prior to initiating the read/writeback.

IOW: If p1 had already grabbed a layout segment covering the area being accessed by p2, then under the old code, we would still have forged ahead and performed the read/write on the DS without calling LAYOUTGET at all.

Quoted 3 lines

How about sending ACCESS for any principal before using a newly retrieved layout at OPEN time or after the layout was revoked/reacquired to simulate the in-band behavior in a practical manner?

If you want to do that for the objects and blocks layout types, then fine, but I see no reason to do it for files layouts: the files DSes will do access checking using the cred passed with the READ/WRITE regardless of what happened with LAYOUTGET.

Quoted 5 lines

Note that I expect some inaccuracies in behavior even with sending ACCESS as
the linux nfs server and other commercial servers bypass permission checking for the file owner at I/O time but not for ACCESS. I believe this was done to simulate (sort of) Posix behavior that allows I/O to an open file even after changing its security attributes.

Quoted 8 lines

Also, do we deal correctly with LAYOUTGET failing on NFS4ERR_ACCESS?
In the example above, if the open order was reversed, LAYOUTGET would have failed for p2's creds as it doesn't have RW access to the file. That would result to reverting to the MDS and the I/O would fail on NFS4ERR_ACCESS as well, yet we'll keep trying (and failing) LAYOUTGET. Optionally, the client could try other creds that opened the file. If the first process to open the file closes it, should we use different creds for LAYOUTGET? With the latest implementation we keep the first opener creds referenced until we return the whole layout, right?

I'm open to the idea of having an NFS4ERR_ACCESS reply to LAYOUTGET fail the entire I/O attempt without an attempt to fail back to MDS.

As for switching creds on close, I believe that is still forbidden under the rules guiding the EXCHGID4_FLAG_BIND_PRINC_STATEID flag (RFC5661, section 18.35.3). Under those rules, a server that sets the EXCHGID4_FLAG_BIND_PRINC_STATEID in the EXCHANGE_ID reply MUST return NFS4ERR_WRONG_CRED in response to the LAYOUTGET call if it tries to authenticate a layout or open stateid that was created by p1, using the principal of p2.

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#2 lore raw
Quoted 23 lines · Trond
Quoted 13 lines · Benny Halevy

Trond, with the latest code for issuing LAYOUTGET with the right credentials we still seem to have a problem with the objects and blocks layout where the security enforcement over out-of-band I/O differs from than the one over in-band I/O.

Consider the following scenario:

file is owned by <uid1, gid1>, mode 660
process p1 successfully opens the file for RW with <uid1, gid1> (client sent OPEN) process p2 successfully opens the file for RW with <uid2, gid1> (client sent ACCESS) client gets a layout using LAYOUTGET for IOMODE_RW
the file is chmod'ed to 600

Quoted 8 lines

now, empirically, in-band I/O would succeed for p1 and fail for p2 (as seen on linux and some commercial servers)

for out-of-band I/O, an object-based server will fence-off the object and recall the layout to enforce the client to refresh its layout, send a LAYOUTGET, reauthorize, and get a new capability. BUT, that's not enough as the new layout and capability, would allow both p1 and p2 access to the object (as the layout is global to the client), yet we want only p1 to have access now.

Quoted 4 lines

I don't understand why you think this is related to the LAYOUTGET credential change. The only difference that the credential change brings is to the case where the client doesn't hold a layout segment prior to initiating the read/writeback.

Correct, this is not caused by that change, just that testing the change raised this issue.

Quoted 6 lines

IOW: If p1 had already grabbed a layout segment covering the area being accessed by p2, then under the old code, we would still have forged ahead and performed the read/write on the DS without calling LAYOUTGET at all.

Quoted 9 lines
Quoted 3 lines

How about sending ACCESS for any principal before using a newly retrieved layout at OPEN time or after the layout was revoked/reacquired to simulate the in-band behavior in a practical manner?

If you want to do that for the objects and blocks layout types, then fine, but I see no reason to do it for files layouts: the files DSes will do access checking using the cred passed with the READ/WRITE regardless of what happened with LAYOUTGET.

True.

Quoted 17 lines
Quoted 5 lines

Note that I expect some inaccuracies in behavior even with sending ACCESS as
the linux nfs server and other commercial servers bypass permission checking for the file owner at I/O time but not for ACCESS. I believe this was done to simulate (sort of) Posix behavior that allows I/O to an open file even after changing its security attributes.

Quoted 8 lines

Also, do we deal correctly with LAYOUTGET failing on NFS4ERR_ACCESS?
In the example above, if the open order was reversed, LAYOUTGET would have failed for p2's creds as it doesn't have RW access to the file. That would result to reverting to the MDS and the I/O would fail on NFS4ERR_ACCESS as well, yet we'll keep trying (and failing) LAYOUTGET. Optionally, the client could try other creds that opened the file. If the first process to open the file closes it, should we use different creds for LAYOUTGET? With the latest implementation we keep the first opener creds referenced until we return the whole layout, right?

I'm open to the idea of having an NFS4ERR_ACCESS reply to LAYOUTGET fail the entire I/O attempt without an attempt to fail back to MDS.

Quoted 9 lines

As for switching creds on close, I believe that is still forbidden under the rules guiding the EXCHGID4_FLAG_BIND_PRINC_STATEID flag (RFC5661, section 18.35.3). Under those rules, a server that sets the EXCHGID4_FLAG_BIND_PRINC_STATEID in the EXCHANGE_ID reply MUST return NFS4ERR_WRONG_CRED in response to the LAYOUTGET call if it tries to authenticate a layout or open stateid that was created by p1, using the principal of p2.

Yeah, that's a bummer.
Even if the client returns the whole layout and tried to retrieve a new layout using the open stateid, it still bound to the original open creds.

Benny