Skip to contentSkip to conversation

NFS/krb and batch jobs - doable?

NFS raini-9HxftnAiGddWk0Htik3J/[email protected] 5 participants 20 messages Oct 9, 2009 – Oct 14, 2009 9 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've been struggling for some time to understand how to allow users of long-running batch jobs to continue to do so in an environmnet migrated to NFS4 with kerberos (from non-kerberized NFS), centered around the problem of keeping their batch job credentials separate from login credentials, which might otherwise mangle or delete them and cause the batch jobs to die on logout - or say if the users opted for renewable tickets for the job explicitly but the system default is non-renewable.

The central problem seems to be that NFS4 (on Linux at least) doesn't support the concept of multiple sessions, and seems to expect to see a standard ccache filename /tmp/krb5cc_${UID}. Am I right in thinking this is a fundamental limitation? I've played with KRB5CCNAME but am led to believe NFS ignores this; I've also added ccname template settings to krb5.conf and PAM to try to fool NFS into separating batch job credentials from user ones, but don't seem to be getting a robust separation.

The essential problem is that I need an environment where a user can obtain credentials that may be different from the desirable interactive default (e.g. longer duration, renewable), then run a batch job, ideally in the heterogeneous ways they do currently - run via a shell, kicked off on another machine via ssh, via condor - and not have its credentials at risk if the user then logs into the machine it's running on, submits a second job etc.

Is this doable currently? I'm surprised (so far) to not run across people who are doing this, and would be very surprised if it's not a requirement for many. If not, is it likely to be doable soon and what does it depend on?

I can supply more technical details of what I've tried if useful. Please also let me know if there's a more appropriate place I can ask these questions. Thanks.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): c8e974302190b867ad8ea49d8158f1db.squirrel-2RFepEojUI30fF+2cCIZ11aTQe2KTcn/@public.gmane.org

#2 lore raw
Quoted 8 lines · On Fri, 9 Oct 2009 08:15:12 -0700 raini-9HxftnAiGddWk0Htik3J/[email protected]…

I've been struggling for some time to understand how to allow users of long-running batch jobs to continue to do so in an environmnet migrated to NFS4 with kerberos (from non-kerberized NFS), centered around the problem of keeping their batch job credentials separate from login credentials, which might otherwise mangle or delete them and cause the batch jobs to die on logout - or say if the users opted for renewable tickets for the job explicitly but the system default is non-renewable.

Quoted 4 lines

The central problem seems to be that NFS4 (on Linux at least) doesn't support the concept of multiple sessions, and seems to expect to see a standard ccache filename /tmp/krb5cc_${UID}. Am I right in thinking this is a fundamental limitation?

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Quoted 5 lines

I've played with KRB5CCNAME but am led to
believe NFS ignores this; I've also added ccname template settings to krb5.conf and PAM to try to fool NFS into separating batch job credentials from user ones, but don't seem to be getting a robust separation.

You're correct that NFS ignores $KRB5CCNAME. It uses the above (less than optimal) heuristic instead.

Quoted 8 lines

The essential problem is that I need an environment where a user can obtain credentials that may be different from the desirable interactive default (e.g. longer duration, renewable), then run a batch job, ideally in the heterogeneous ways they do currently - run via a shell, kicked off on another machine via ssh, via condor - and not have its credentials at risk if the user then logs into the machine it's running on, submits a second job etc.

Quoted 5 lines

Is this doable currently? I'm surprised (so far) to not run across people who are doing this, and would be very surprised if it's not a requirement for many. If not, is it likely to be doable soon and what does it depend on?

Probably doable, but not trivial. IIRC, the kernel tracks credentials by uid. You'd need to determine some way to split that up so that each "session" has separate credentials. Once you do that, you'll have to have the kernel pass enough info to the upcall for it to determine what credcache it should use and modify gssd to use the new info accordingly.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091009121602.5ec86dfb-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#4 lore raw
Quoted 6 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

You're correct that NFS ignores $KRB5CCNAME. It uses the above (less than optimal) heuristic instead.

Thanks for explaining this Jeff - this does accord with what I see - which of course leaves my batch job system unpredictable.

Quoted 5 lines

Probably doable, but not trivial. IIRC, the kernel tracks credentials by uid. You'd need to determine some way to split that up so that each "session" has separate credentials. Once you do that, you'll have to have the kernel pass enough info to the upcall for it to determine what credcache it should use and modify gssd to use the new info accordingly.

Just to be clear - you mean doable to a coder who might like to improve on gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

A message that is not in the archive (sent off-list or not yet received): 1c358fde92c49215d84129a1bfe2c6ec.squirrel-2RFepEojUI30fF+2cCIZ11aTQe2KTcn/@public.gmane.org

#6 lore raw
Quoted 16 lines · On Fri, 9 Oct 2009 09:53:51 -0700 raini-9HxftnAiGddWk0Htik3J/[email protected]…
Quoted 6 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

You're correct that NFS ignores $KRB5CCNAME. It uses the above (less than optimal) heuristic instead.

Thanks for explaining this Jeff - this does accord with what I see - which of course leaves my batch job system unpredictable.

Quoted 5 lines

Probably doable, but not trivial. IIRC, the kernel tracks credentials by uid. You'd need to determine some way to split that up so that each "session" has separate credentials. Once you do that, you'll have to have the kernel pass enough info to the upcall for it to determine what credcache it should use and modify gssd to use the new info accordingly.

Quoted 4 lines

Just to be clear - you mean doable to a coder who might like to improve on gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd code.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091010090039.4dfd1dfb-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#8 lore raw
Quoted 8 lines · Jeff Layton
Quoted 5 lines

Just to be clear - you mean doable to a coder who might like to improve on
gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd code.

Thanks for confirming. Skipping back a little:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mtime rather than actually scanning /tmp/krb5cc_uid* for ccache files with the latest TGT expiration, which is how I originally read your statement. This seemingly would make a difference in an environment with a batch job with a long lifetime ticket and subsequent interactive login generating a separate ccache file with a shorter lifetime but newer mtime.

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems* to me it only looks at mtime, although what you suggest would be more optimal. Could you confirm whether it's scanning ccache files for longest TGT, or just using mtime?

A message that is not in the archive (sent off-list or not yet received): ee56329e7d86a3e4b15001a39bb7e14a.squirrel-2RFepEojUI30fF+2cCIZ11aTQe2KTcn/@public.gmane.org

#10 lore raw
Quoted 23 lines · On Tue, 13 Oct 2009 08:28:52 -0700 raini-9HxftnAiGddWk0Htik3J/[email protected]…
Quoted 8 lines · Jeff Layton
Quoted 5 lines

Just to be clear - you mean doable to a coder who might like to improve on
gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd code.

Thanks for confirming. Skipping back a little:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mtime rather than actually scanning /tmp/krb5cc_uid* for ccache files with the latest TGT expiration, which is how I originally read your statement. This seemingly would make a difference in an environment with a batch job with a long lifetime ticket and subsequent interactive login generating a separate ccache file with a shorter lifetime but newer mtime.

Quoted 5 lines

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems* to me it only looks at mtime, although what you suggest would be more optimal. Could you confirm whether it's scanning ccache files for longest TGT, or just using mtime?

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091013114441.2882c8b9-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#12 lore raw

On Tue, Oct 13, 2009 at 11:44 AM, Jeff Layton <[email protected]> wrot= e:

Quoted 5 lines
Quoted 2 lines · On Tue, 13 Oct 2009 08:28:52 -0700 raini-9HxftnAiGddWk0Htik3J/[email protected]…
Quoted 1 line · Jeff Layton
Quoted 1 line

Just to be clear - you mean doable to a coder who might like to i=

mprove

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

on
gssd/kernel credential separation, rather than a non-coding sysad=

min who

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.g=

ssd

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

code.

Thanks for confirming. =A0Skipping back a little:

Quoted 1 line
Quoted 1 line
Quoted 1 line

No, gssd (the client side daemon) will search /tmp for anything=

that

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mti=

me

Quoted 1 line
Quoted 1 line

rather than actually scanning /tmp/krb5cc_uid* for ccache files with=

the

Quoted 1 line
Quoted 1 line

latest TGT expiration, which is how I originally read your statement=

=2E

Quoted 1 line
Quoted 1 line

This seemingly would make a difference in an environment with a batc=

h job

Quoted 1 line
Quoted 1 line

with a long lifetime ticket and subsequent interactive login generat=

ing a

Quoted 3 lines
Quoted 3 lines

separate ccache file with a shorter lifetime but newer mtime.

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *=

seems*

Quoted 1 line
Quoted 1 line

to me it only looks at mtime, although what you suggest would be mor=

e

Quoted 1 line
Quoted 1 line

optimal. =A0Could you confirm whether it's scanning ccache files for=

longest

Quoted 4 lines
Quoted 2 lines

TGT, or just using mtime?

You and Kevin are correct. rpc.gssd only looks at the mtime. When I d=

id

Quoted 1 line

the work to allow the CIFS SPNGEO upcall to find alternate credcaches=

,

Quoted 8 lines

I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

I agree it shouldn't be too hard to change if that behavior is desirabl= e/useful.

#13 lore raw
Quoted 38 lines · Kevin Coffman
Quoted 37 lines · Jeff Layton
Quoted 23 lines · On Tue, 13 Oct 2009 08:28:52 -0700 [email protected] wrote:
Quoted 8 lines · Jeff Layton
Quoted 5 lines

Just to be clear - you mean doable to a coder who might like to improve on
gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd code.

Thanks for confirming. Skipping back a little:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mtime rather than actually scanning /tmp/krb5cc_uid* for ccache files with the latest TGT expiration, which is how I originally read your statement. This seemingly would make a difference in an environment with a batch job with a long lifetime ticket and subsequent interactive login generating a separate ccache file with a shorter lifetime but newer mtime.

Quoted 5 lines

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems* to me it only looks at mtime, although what you suggest would be more optimal. Could you confirm whether it's scanning ccache files for longest TGT, or just using mtime?

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

Quoted 7 lines
Quoted 5 lines

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

I agree it shouldn't be too hard to change if that behavior is desirable/useful.

Shouldn't rpc.gssd in any case be looking for all _valid_ credcaches, rather than just any old credcache-like file?

If you stick to that principle, then if there is one file that contains a still-unexpired TGT, and one with a TGT that has expired, then the first file should always be preferred, irrespective of the value of the mtimes...

Cheers
Trond

#14 lore raw
Quoted 47 lines · On Tue, 13 Oct 2009 12:56:20 -0400 Trond Myklebust…
Quoted 38 lines · Kevin Coffman
Quoted 37 lines · Jeff Layton
Quoted 23 lines · On Tue, 13 Oct 2009 08:28:52 -0700 [email protected] wrote:
Quoted 8 lines · Jeff Layton
Quoted 5 lines

Just to be clear - you mean doable to a coder who might like to improve on
gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd code.

Thanks for confirming. Skipping back a little:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mtime rather than actually scanning /tmp/krb5cc_uid* for ccache files with the latest TGT expiration, which is how I originally read your statement. This seemingly would make a difference in an environment with a batch job with a long lifetime ticket and subsequent interactive login generating a separate ccache file with a shorter lifetime but newer mtime.

Quoted 5 lines

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems* to me it only looks at mtime, although what you suggest would be more optimal. Could you confirm whether it's scanning ccache files for longest TGT, or just using mtime?

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

Quoted 7 lines
Quoted 5 lines

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

I agree it shouldn't be too hard to change if that behavior is desirable/useful.

Quoted 8 lines

Shouldn't rpc.gssd in any case be looking for all _valid_ credcaches, rather than just any old credcache-like file?

If you stick to that principle, then if there is one file that contains a still-unexpired TGT, and one with a TGT that has expired, then the first file should always be preferred, irrespective of the value of the mtimes...

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091013132701.72927b4d-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#16 lore raw
Quoted 7 lines · Jeff Layton

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

So why do you think that is a problem? The result should be that rpc.gssd always ends up with a valid credential as long as there is at least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long as the RPC client can always instantiate a new one.

Trond

#17 lore raw
Quoted 9 lines · On Tue, 13 Oct 2009 13:51:33 -0400 Trond Myklebust…
Quoted 7 lines · Jeff Layton

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

Quoted 6 lines

So why do you think that is a problem? The result should be that rpc.gssd always ends up with a valid credential as long as there is at least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long as the RPC client can always instantiate a new one.

Hrm...good point. I suppose that as long as gssd can pick a new credcache if the context expires then this patch is superfluous. Wasn't that support only added fairly recently (around a year ago?)? If so, it may just be that raini isn't using a recent enough nfs-utils...

Signature · 1 line
Jeff Layton <[email protected]>
#18 lore raw
Quoted 19 lines
Quoted 9 lines · On Tue, 13 Oct 2009 13:51:33 -0400 Trond Myklebust…
Quoted 7 lines · Jeff Layton

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

Quoted 6 lines

So why do you think that is a problem? The result should be that rpc.gssd always ends up with a valid credential as long as there is at least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long as the RPC client can always instantiate a new one.

Quoted 4 lines

Hrm...good point. I suppose that as long as gssd can pick a new credcache if the context expires then this patch is superfluous. Wasn't that support only added fairly recently (around a year ago?)? If so, it may just be that raini isn't using a recent enough nfs-utils...

Hm - well I'm stuck on production machines (RHEL5) so currently on nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic either way. Could someone point me to information on this change (I see little in http://www.kernel.org/pub/linux/utils/nfs/)?

The reason I thought the new code would be useful is that if default tickets are non-renewable and short lifetime, it seems sensible for gssd to spot and use a longer lifetime renewable ticket in another ccache file
- and say use krenew to keep the job alive (or even cope with the user renewing the ticket manually).

Seems to me therefore that in the absence of per-session ccaches, gssd should prefer long lifetime, and renewable.

Would the newer code you mention cope with this situation already?

#19 lore raw
Quoted 29 lines · On Wed, 2009-10-14 at 09:47 -0700, [email protected] wrote:
Quoted 19 lines
Quoted 9 lines · On Tue, 13 Oct 2009 13:51:33 -0400 Trond Myklebust…
Quoted 7 lines · Jeff Layton

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

Quoted 6 lines

So why do you think that is a problem? The result should be that rpc.gssd always ends up with a valid credential as long as there is at least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long as the RPC client can always instantiate a new one.

Quoted 4 lines

Hrm...good point. I suppose that as long as gssd can pick a new credcache if the context expires then this patch is superfluous. Wasn't that support only added fairly recently (around a year ago?)? If so, it may just be that raini isn't using a recent enough nfs-utils...

Hm - well I'm stuck on production machines (RHEL5) so currently on nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic either way. Could someone point me to information on this change (I see little in http://www.kernel.org/pub/linux/utils/nfs/)?

Quoted 8 lines

The reason I thought the new code would be useful is that if default tickets are non-renewable and short lifetime, it seems sensible for gssd to spot and use a longer lifetime renewable ticket in another ccache file
- and say use krenew to keep the job alive (or even cope with the user renewing the ticket manually).

Seems to me therefore that in the absence of per-session ccaches, gssd should prefer long lifetime, and renewable.

No. That is not a necessary condition.

Quoted 1 line

Would the newer code you mention cope with this situation already?

Yes. Please reread the thread carefully again.

As long as gssd always select from among the valid (i.e. unexpired) TGTs, then the actual remaining lifetime of the particular TGT that was selected doesn't matter; gssd will always be able to renew the resulting RPCSEC_GSS session when it expires.

Trond

#20 lore raw
In reply to [email protected]
Quoted 35 lines · On Wed, Oct 14, 2009 at 12:47 PM, <[email protected]> wrote:
Quoted 19 lines
Quoted 9 lines · On Tue, 13 Oct 2009 13:51:33 -0400 Trond Myklebust…
Quoted 7 lines · Jeff Layton

Correct...and gssd actually does check the validity of the cache. If TGT has expired or it's not valid for some other reason, then it skips it and moves on.

The problem comes when you have more than one valid credcache. In that case it picks the one with the latest mtime. It seems that it should instead pick the one with the latest TGT expiration time.

Quoted 6 lines

So why do you think that is a problem? The result should be that rpc.gssd always ends up with a valid credential as long as there is at least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long as the RPC client can always instantiate a new one.

Quoted 4 lines

Hrm...good point. I suppose that as long as gssd can pick a new credcache if the context expires then this patch is superfluous. Wasn't that support only added fairly recently (around a year ago?)? If so, it may just be that raini isn't using a recent enough nfs-utils...

Hm - well I'm stuck on production machines (RHEL5) so currently on nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic either way.  Could someone point me to information on this change (I see little in http://www.kernel.org/pub/linux/utils/nfs/)?

The reason I thought the new code would be useful is that if default tickets are non-renewable and short lifetime, it seems sensible for gssd to spot and use a longer lifetime renewable ticket in another ccache file
- and say use krenew to keep the job alive (or even cope with the user renewing the ticket manually).

Quoted 4 lines

Seems to me therefore that in the absence of per-session ccaches, gssd should prefer long lifetime, and renewable.

Would the newer code you mention cope with this situation already?

The change that adds the check for "valid" credentials caches (including expiration) was not added until nfs-utils-1.1.3.

With that version of rpc.gssd, Jeff and Trond's descriptions are correct.

K.C.

#21 lore raw
In reply to Jeff Layton
Quoted 11 lines

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

This would be a big plus from me - I still wouldn't be able to create per-job ccaches of course, but if a user who knew they needed to run a job could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch, say, and NFS would use this in preference to a later login ticket, this would really help.

#22 lore raw
Quoted 10 lines · On Tue, 13 Oct 2009 08:59:29 -0700 [email protected] wrote:
Quoted 10 lines

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Quoted 9 lines
Quoted 1 line

Kevin, any thoughts?

This would be a big plus from me - I still wouldn't be able to create per-job ccaches of course, but if a user who knew they needed to run a job could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch, say, and NFS would use this in preference to a later login ticket, this would really help.

Ok, here's a proposed patch...only compile-tested so far. I don't have time at the moment to test it more extensively so if you could test it out and report back, that would be helpful.

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>
Quoted 1 line

From 39564369e8a22b97f4a3021eb527eaf6524ca557 Mon Sep 17 00:00:00 2001

From: Jeff Layton <[email protected]>
Date: Tue, 13 Oct 2009 13:27:52 -0400
Subject: [PATCH] gssd: prefer credcaches with latest TGT expiration

(Note: this is only compile-tested so far)

gssd currently walks all of the entities in the credcache dir that look like credcaches, verifies that they are valid and picks the one with the latest mtime.

There's a problem here however, the one with the latest mtime might still expire very soon after we pick it. Instead of doing that, have it instead pick the credcache that has the latest TGT expiration.

With this, someone who wants to run a long-running batch job can get a TGT with a long expiration time and have some reasonable hope that it'll run to completion.

---
 utils/gssd/krb5_util.c |   72 ++++++++++++++++++-----------------------------
 1 files changed, 28 insertions(+), 44 deletions(-)
utils/gssd/krb5_util.c +28 -44
index 78e9775..8f1f56d 100644
@@ -134,11 +134,11 @@ struct gssd_k5_kt_princ *gssd_k5_kt_princ_list = NULL;
 /*==========================*/
 
 static int select_krb5_ccache(const struct dirent *d);
-static int gssd_find_existing_krb5_ccache(uid_t uid, char *dirname,
+static time_t gssd_find_existing_krb5_ccache(uid_t uid, char *dirname,
 		struct dirent **d);
 static int gssd_get_single_krb5_cred(krb5_context context,
 		krb5_keytab kt, struct gssd_k5_kt_princ *ple, int nocache);
-static int query_krb5_ccache(const char* cred_cache, char **ret_princname,
+static time_t query_krb5_ccache(const char* cred_cache, char **ret_princname,
 		char **ret_realm);
 
 /*
@@ -171,24 +171,22 @@ select_krb5_ccache(const struct dirent *d)
  * The caller is responsible for freeing the dirent if one is returned.
  *
  * Returns:
- *	0 => could not find an existing entry
- *	1 => found an existing entry
+ *	 0 => could not find an existing entry
+ *	>0 => found an existing entry
  */
-static int
+static time_t
 gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 {
 	struct dirent **namelist;
 	int n;
 	int i;
-	int found = 0;
 	struct dirent *best_match_dir = NULL;
-	struct stat best_match_stat, tmp_stat;
+	struct stat tmp_stat;
 	char buf[1030];
 	char *princname = NULL;
 	char *realm = NULL;
-	int score, best_match_score = 0;
+	time_t tmp_time, best_match_time = 0;
 
-	memset(&best_match_stat, 0, sizeof(best_match_stat));
 	*d = NULL;
 	n = scandir(dirname, &namelist, select_krb5_ccache, 0);
 	if (n < 0) {
@@ -225,18 +223,15 @@ gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 				free(namelist[i]);
 				continue;
 			}
-			if (!query_krb5_ccache(buf, &princname, &realm)) {
+
+			tmp_time = query_krb5_ccache(buf, &princname, &realm);
+			if (!tmp_time) {
 				printerr(3, "CC file '%s' is expired or corrupt\n",
 					 statname);
 				free(namelist[i]);
 				continue;
 			}
 
-			score = 0;
-			if (preferred_realm &&
-					strcmp(realm, preferred_realm) == 0) 
-				score++;
-
 			printerr(3, "CC file '%s'(%s@%s) passed all checks and"
 				    " has mtime of %u\n",
 				 statname, princname, realm, 
@@ -246,49 +241,38 @@ gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 			 * recent (the one with the latest mtime), and
 			 * don't free the dirent
 			 */
-			if (!found) {
+			if (!best_match_time) {
 				best_match_dir = namelist[i];
-				best_match_stat = tmp_stat;
-				best_match_score = score;
-				found++;
-			}
-			else {
+				best_match_time = tmp_time;
+			} else {
 				/*
-				 * If current score is higher than best match 
-				 * score, we use the current match. Otherwise,
-				 * if the current match has an mtime later
-				 * than the one we are looking at, then use
-				 * the current match.  Otherwise, we still
-				 * have the best match.
+				 * prefer the credcache that has the latest
+				 * TGT expiration time. This is helpful when
+				 * for ensuring that long-running jobs don't
+				 * lose their credentials.
 				 */
-				if (best_match_score < score ||
-				    (best_match_score == score && 
-				       tmp_stat.st_mtime >
-					    best_match_stat.st_mtime)) {
+				if (tmp_time > best_match_time) {
 					free(best_match_dir);
 					best_match_dir = namelist[i];
-					best_match_stat = tmp_stat;
-					best_match_score = score;
-				}
-				else {
+					best_match_time = tmp_time;
+				} else {
 					free(namelist[i]);
 				}
 				printerr(3, "CC file '%s/%s' is our "
 					    "current best match "
-					    "with mtime of %u\n",
+					    "with expire time of %u\n",
 					 dirname, best_match_dir->d_name,
-					 best_match_stat.st_mtime);
+					 best_match_time);
 			}
 			free(princname);
 			free(realm);
 		}
 		free(namelist);
 	}
-	if (found)
-	{
+	if (best_match_time)
 		*d = best_match_dir;
-	}
-	return found;
+
+	return best_match_time;
 }
 
 
@@ -935,7 +919,7 @@ static inline int data_is_equal(krb5_data d1, krb5_data d2)
 		&& memcmp(d1.data, d2.data, d1.length) == 0);
 }
 
-static int
+static time_t
 check_for_tgt(krb5_context context, krb5_ccache ccache,
 	      krb5_principal principal)
 {
@@ -959,7 +943,7 @@ check_for_tgt(krb5_context context, krb5_ccache ccache,
 				data_is_equal(creds.server->data[1],
 					      principal->realm) &&
 				creds.times.endtime > time(NULL))
-			found = 1;
+			found = creds.times.endtime;
 		krb5_free_cred_contents(context, &creds);
 	}
 	krb5_cc_end_seq_get(context, ccache, &cur);
@@ -967,7 +951,7 @@ check_for_tgt(krb5_context context, krb5_ccache ccache,
 	return found;
 }
 
-static int
+static time_t
 query_krb5_ccache(const char* cred_cache, char **ret_princname,
 		  char **ret_realm)
 {
Signature · 1 line
1.6.0.6
#23 lore raw
Quoted 23 lines · On Tue, 13 Oct 2009 13:31:38 -0400 Jeff Layton <[email protected]> wrote:
Quoted 10 lines · On Tue, 13 Oct 2009 08:59:29 -0700 [email protected] wrote:
Quoted 10 lines

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Quoted 9 lines
Quoted 1 line

Kevin, any thoughts?

This would be a big plus from me - I still wouldn't be able to create per-job ccaches of course, but if a user who knew they needed to run a job could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch, say, and NFS would use this in preference to a later login ticket, this would really help.

Quoted 4 lines

Ok, here's a proposed patch...only compile-tested so far. I don't have time at the moment to test it more extensively so if you could test it out and report back, that would be helpful.

Looks like this patch will probably break the "preferred realm" code. It'll have to be respun to fix that, but it should work as expected in a single-realm environment.

Signature · 1 line
Jeff Layton <[email protected]>
#24 lore raw
In reply to Jeff Layton
Quoted 23 lines
Quoted 12 lines · On Tue, 13 Oct 2009 08:59:29 -0700 [email protected] wrote:
Quoted 1 line

You and Kevin are correct. rpc.gssd only looks at the mtime. When I

did

Quoted 9 lines

the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

Quoted 7 lines

This would be a big plus from me - I still wouldn't be able to create per-job ccaches of course, but if a user who knew they needed to run a job
could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch,
say, and NFS would use this in preference to a later login ticket, this would really help.

Quoted 3 lines

Ok, here's a proposed patch...only compile-tested so far. I don't have time at the moment to test it more extensively so if you could test it out and report back, that would be helpful.

Thanks Jeff - this looks extremely useful, caveat my other comment (and perhaps lack of understanding) on the list today about what's happened in recent nfs-utils which I'd like to clarify.

I may have trouble testing this in the short term as I'm largely bound to production environments - but will get to back to you if I can.

#25 lore raw
Quoted 27 lines · On Wed, 14 Oct 2009 10:00:59 -0700 [email protected] wrote:
Quoted 23 lines
Quoted 12 lines · On Tue, 13 Oct 2009 08:59:29 -0700 [email protected] wrote:
Quoted 1 line

You and Kevin are correct. rpc.gssd only looks at the mtime. When I

did

Quoted 9 lines

the work to allow the CIFS SPNGEO upcall to find alternate credcaches, I implemented the behavior I described (prefer the latest TGT expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer credcaches with the latest TGT expiration if it was considered a desirable change.

Kevin, any thoughts?

Quoted 7 lines

This would be a big plus from me - I still wouldn't be able to create per-job ccaches of course, but if a user who knew they needed to run a job
could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch,
say, and NFS would use this in preference to a later login ticket, this would really help.

Quoted 3 lines

Ok, here's a proposed patch...only compile-tested so far. I don't have time at the moment to test it more extensively so if you could test it out and report back, that would be helpful.

Quoted 7 lines

Thanks Jeff - this looks extremely useful, caveat my other comment (and perhaps lack of understanding) on the list today about what's happened in recent nfs-utils which I'd like to clarify.

I may have trouble testing this in the short term as I'm largely bound to production environments - but will get to back to you if I can.

Actually...I'm not convinced that it is that useful. As Trond pointed out, when the credentials expire, the kernel should upcall for new creds. As long as there is a valid TGT in a credcache for that user somewhere then it should just pick up that one and keep humming along. If that's not working for some reason then that's likely a bug.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): f99e65f7b2fe66fc32dee931fd6bd525.squirrel-2RFepEojUI30fF+2cCIZ11aTQe2KTcn/@public.gmane.org

#28 lore raw
Quoted 16 lines
Quoted 6 lines

No, gssd (the client side daemon) will search /tmp for anything that looks like a credcache for the right user, verify that it is a credcache and then pick the one with the latest TGT expiration.

You're correct that NFS ignores $KRB5CCNAME. It uses the above (less than optimal) heuristic instead.

Thanks for explaining this Jeff - this does accord with what I see - which of course leaves my batch job system unpredictable.

Quoted 5 lines

Probably doable, but not trivial. IIRC, the kernel tracks credentials by uid. You'd need to determine some way to split that up so that each "session" has separate credentials. Once you do that, you'll have to have the kernel pass enough info to the upcall for it to determine what credcache it should use and modify gssd to use the new info accordingly.

Quoted 4 lines

Just to be clear - you mean doable to a coder who might like to improve on gssd/kernel credential separation, rather than a non-coding sysadmin who needs with work within the current NFS/gssd framework?

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs