Don't hang user processes if Kerberos ticket for nfs4 mount expires

With recent kernels if the Kerberos ticket for a nfs4 mount expires any user process trying to access the mount hangs until a new ticket is obtained. Simultaneously a (luckily rate-limited, but still seemingly endless) stream of "Error: state manager encountered RPCSEC_GSS session expired against NFSv4 server" messages is written to the kernel log.

In a common setup with user home directories nfs4 mounted on workstations one of the processes that is likely to hang is the screen-unlock function which would normally (via pam_krb5 or similar) get the new ticket.

In older kernels the EKEYEXPIRED error would be passed to userland, which would usualy just give up.

This patch restores the old behavior, which makes nfs4 mounted home directories usable for me.

index 4700fae..dc28a78 100644
@@ -302,7 +302,6 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struc
 			}
 		case -NFS4ERR_GRACE:
 		case -NFS4ERR_DELAY:
-		case -EKEYEXPIRED:
 			ret = nfs4_delay(server->client, &exception->timeout);
 			if (ret != 0)
 				break;

@@ -3732,7 +3731,6 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
 		case -NFS4ERR_DELAY:
 			nfs_inc_server_stats(server, NFSIOS_DELAY);
 		case -NFS4ERR_GRACE:
-		case -EKEYEXPIRED:
 			rpc_delay(task, NFS4_POLL_RETRY_MAX);
 			task->tk_status = 0;
 			return -EAGAIN;

index 39914be..2bee41e 100644
@@ -1377,8 +1377,9 @@ static int nfs4_recovery_handle_error(struct nfs_client *clp, int error)
 			/* Zero session reset errors */
 			return 0;
 		case -EKEYEXPIRED:
-			/* Nothing we can do */
-			nfs4_warn_keyexpired(clp->cl_hostname);
+			/* Nothing we can do, so do nothing.  Don't even
+			   print a warning message, this is not a kernel
+			   problem */
 			return 0;
 	}
 	return error;

Uhhh, no...EKEYEXPIRED was never passed to userland. The patchset that added EKEYEXPIRED returns in this codepath also added the code to make it hang.

This not a bug, or at least it's intentional behavior. When a krb5 ticket expires, we *want* the process to hang. Otherwise, people with long running jobs will often find that their jobs error out inexplicably when their ticket expires.

The patches that introduced this behavior went into 2.6.34. See the commits around 2c64348 (and some preceding ones in the rpc layer).

If you want to fix this use case, you'll need to come up with a scheme that doesn't regress this behavior. I think that you'll really need to ensure that whatever process you expect to re-fetch your TGT is not dependent on accessing kerberized nfs mounts. That really seems like an untenable chicken and egg situation.

Jeff Layton <[email protected]>

Jeff Layton wrote:

Uhhh, no...EKEYEXPIRED was never passed to userland. The patchset that added EKEYEXPIRED returns in this codepath also added the code to make it hang.

This not a bug, or at least it's intentional behavior. When a krb5 ticket expires, we *want* the process to hang. Otherwise, people with long running jobs will often find that their jobs error out inexplicably when their ticket expires.

Who decided that? This seems completely wrong to me. If my credentials expire, I want to get permission denied, not a client hang. In 20 years of using authenticated file systems I never once wished my process had hung when my ticket expired.

Why should this be any different from any other failure condition? If you try to open a file that doesn't exist, do you want your process to hang instead of getting ENOENT, just in case the file magically appears at some point in the future?

This seems a recipe for disaster. Suppose I have a cron job that fires once a minute, and all those jobs hang waiting for a ticket. I come to work in the morning and discover I've got 10,000 hung processes. Or not, because my computer has crashed from resource exhaustion.

I proposed it, we discussed it on the list, and Trond and Steve committed the patches necessary to make it happen. This was back in late 2009/early 2010 though, so my memory is a bit fuzzy...

That's different. Not renewing your credentials is often a temporary situation. Kerberos is different than other authentication methods in that you get a ticket only for a period of time, so expired credentials are not a situation that's common with other authentication methods.

The previous situation was also a recipe for disaster, and was often cited as a primary reason why people didn't want to deploy kerberized NFS. Having everything fall down and go boom when your ticket expires is not desirable either.

I suppose we'll have to agree to disagree on this point. That said, I'm open to sane suggestions however that don't regress the behavior for those users who need to be able to cope with expired tickets.

Jeff Layton <[email protected]>

Note too that the gssd code distinguishes between an expired TGT and a non-existent credcache. The latter will give you the error you desire here. So one possibility is just to remove the credcache from /tmp in this situation.

Another possibility might be a new option to rpc.gssd that allows the user to select the error that it passes back to the kernel on an expired ticket.

Jeff Layton <[email protected]>

Hi,

While I'm not expert in this area, my impression had been that the established practice was that used with AFS, i.e., run jobs under a process capable of renewing kerberos tickets, e.g., kstart (http://www.eyrie.org/~eagle/software/kstart/).

Matt

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

[...]

You are, of course, right. userland used to get EPERM.

I thought that was what kstart/krenew were for.

Ah, I'm a Debian user - 2.6.32 for the moment, soon to be 3.?

Ow. "Fixing" (at least) Gnome-3 and Gnome-2 screen-lock/screensavers.

How about a mount option to chose between the two behaviours?

Something to scan /tmp for expired credentials and zap em? rpc.gssd would communicate that to the kernel?

Whadaya know, that works.

With the 3.1-rc10 kernel I let my ticket expire, did a ls - it hangs.

Now, from another terminal I do a kdestroy on my ticket cache, and (a second or so later) the ls gets an EPERM.

So this behaviour can be changed from userland with no changes to the kernel, rpc.gssd or anything else.

Some fun racing possibilities.

Here's a dumb perl script that could be run from, for example, .xsession to automatically destroy expired ticket caches.

Would need a bit of trickery to make it go away on end of session and something in /etc/pm/sleep.d to send it a SIGALRM when the system wakes from suspend or hibernate.

It has a potential race between destroying an expired ticket and a new ticket being granted.

I guess now I'll look at a hack to rpc.gssd for a neater way of doing this.

#! /usr/bin/perl -w

my $ALARMED = 0;

$SIG{ALRM} = sub { ++$ALARMED; };

use POSIX qw(mktime);

# Work out ticket expiry

# Valid starting     Expires            Service principal
# 11/17/11 10:34:23  11/17/11 20:34:23  krbtgt/[email protected]

# renew until 11/18/11 10:34:23
# 11/17/11 10:34:23 11/17/11 20:34:23 nfs/[email protected]
# renew until 11/18/11 10:34:23
# 11/17/11 11:24:24 11/17/11 20:34:23 host/[email protected]
# renew until 11/18/11 10:34:23

# Eurgh - non localised, US format dates.

sub expiry {
	local *KLIST;
	open KLIST, "/usr/bin/klist | " or return;
	my $expiry;
	while (<KLIST>) {
		if (m((\d+)/(\d+)/(\d+) (\d+):(\d+):(\d+)  krbtgt)) {
			$expiry = mktime ($6, $5, $4, $2, $1 - 1, 100 +  $3);
			last;
		}
	}

	$expiry;
}

for (;;) {
	my $sleepytime = 60;

	my $expiry = expiry ();
	
	if (defined $expiry) {
		my $left = $expiry - time;
		if ($left <= 0) {
			# Ticket expired, zap it.  Potential race with
			# new ticket creation.
			print "Destroy expired ticket\n";
			system "/usr/bin/kdestroy";
		}
		else {
			$sleepytime = $left;
		}
	}

	if ($ALARMED) {
		$ALARMED = 0;
		next;
	}

	# If machine freezes during this sleap how long will
	# it sleep for?
	print "Sleeping for $sleepytime seconds\n";
	sleep $sleepytime;
}

Ok, I can remember a bit more about the genesis of this scheme...

At the time the argument went something like this:

No one expects that when their krb5 ticket expires that their applications will fail. A case in point is something like a krb5 ssh session. If I had a valid ticket when I initiated the session, then it we would consider it a bug if it were to suddenly die when the ticket expired.

Contrast that however with applications running on a kerberized NFS mount. As soon as the ticket expires they start failing with non-transient errors. This is probably the case as well with screen locker you're using, but it's apparently able to recover enough to allow the TGT to be renewed. I expect though, that you may have other less visible programs that are dying in this situation or are getting unexpected errors.

The current behavior was really intended as a first approximation. I fully expected that it would need some refinement, but AFAIK, no one has complained loudly about the current behavior until now, so I haven't seen need to mess with it.

I'm not that familiar with kstart, but I assume that it gets a renewable TGT and just renews it as needed? I have to wonder if that sort of tool might be verboten in security conscious sites (the very sort that want kerberized nfs).

If we decide that making this behavior switchable is the right thing to do, then what you'll probably want to do is add a new command-line option to rpc.gssd, and make it conditionally return -EKEYEXPIRED or -EACCES in the downcall based on it. It should be a fairly simple patch. See process_krb5_upcall() in rpc.gssd...

Long term, we probably need to consider this use-case in the GSSAPI proxy initiative that Simo has been scoping out. It would be nice to have a solution that would work for both home directory configurations and long-running jobs without needing these sorts of hacks.

Jeff Layton <[email protected]>

I would argue that if you don't want your applications to stop working when your ticket expires, you shouldn't let the ticket expire. If you don't want to have to renew your ticket, you should use an infinite ticket lifetime.

It sounds like you've made up your mind, but I would urge you to make this a mount option, analogous to the hard/soft mount option.

That's the ideal situation, but shit happens, and losing a long-running job can often be an expensive proposition.

I've not made up my mind about anything, and in any case it's not my decision to make. I think you need to convince Trond here... :)

I'm quite open to sane proposals as long as we can accomodate those who are dependent on the current behavior. As I said before, when I originally did the patches a couple of years ago, I sort of figured the current behavior was a first approximation.

A mount option will be harder to implement than a rpc.gssd command-line option, but it sounds reasonable. Still, it would be better not to have to make this an either/or decision somehow.

Jeff Layton <[email protected]>

I'm not arguing that the client machine should hang when tickets expire. I'm arguing that user processes should not hang when tickets expire. Sorry if that wasn't clear.